Imaging device and playback device

ABSTRACT

An imaging device includes an imaging element that acquires a first image based on signal charge generated during a first accumulation time, and a second image based on signal charge generated during a second accumulation time relatively longer than the first accumulation time and synchronized with the first image during a synchronization period including the first accumulation time, and a moving image file generating unit that generates a moving image file including a first moving image based on the first image, a second moving image based on the second image, and synchronization information for synchronizing the first moving image and the second moving image frame by frame.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/258,564, filed on Sep. 7, 2016, which claims the benefit of andpriority to Japanese Patent Application Nos. 2015-177587 and2016-124714, filed on Sep. 9, 2015 and Jun. 23, 2016, respectively, theentire contents of each of which are hereby incorporated by referenceherein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an imaging device having an imageplayback function, and a playback device.

Description of the Related Art

If a moving image and a still image can be shot at the same time withone camera, not only can a shooting scene be viewed as a moving image,but also a decisive scene in the moving image can be seen as a stillimage for fun. This can significantly enhance the values of shot images.Further, if a moving image at a normal frame rate and a moving image ata high frame rate can be shot at the same time with one camera, aspecific scene can be switched to a slow-motion moving image to enjoythe image as a high-definition moving image. This can give a viewer anuplifting feeling.

In the meantime, when a phenomenon, so-called jerkiness, like a kind offrame-by-frame advance happens to a moving image played back, it iscommon that the quality of the moving image is largely degraded. Inorder to suppress the jerkiness, there is a need to set an accumulationtime close to one frame period in a series of shooting processes. Inother words, if the frame rate is 30 fps, a relatively longeraccumulation time, such as 1/30 second or 1/60 second, will be adequate.Particularly, in such a situation that the attitude of a camera isinstable such as a helicopter shot, this setting is important.

On the other hand, since a still image is required to have the sharpnessof shooting a moment, there is a need to set a short accumulation time,for example, about 1/1000 second, in order to obtain a stop motioneffect. Further, in the case of a moving image at a high frame rate, oneframe period is short. Therefore, for example, when the frame rate is120 fps, a short accumulation time such as 1/125 second or 1/250 secondis inevitably set.

Shooting two images at the same time through a single photographic lens,such as a moving image and a still image, or a moving image at a normalframe rate and a moving image at a high frame rate, means that theaperture values used to shooting these images are the same. Even in thiscase, it is desired that similar levels of signal charge should beobtained in an imaging element while shooting two images in differentaccumulation time settings to obtain noiseless images having excellentS/N ratios.

Japanese Patent Application Laid-Open No. 2014-048459 discloses animaging device including a pair of photodiodes having the shape ofpupils asymmetric with respect to each pixel. In the imaging devicedescribed in Japanese Patent Application Laid-Open No. 2014-048459, thelight-receiving efficiency of one of the pair of photodiodes is high andthe light-receiving efficiency of the other photodiode is low. Twosignals from the pair of photodiodes are used as separate pieces ofimage data so that the two images can be shot at the same time.

Further, Japanese Patent Application Laid-Open No. 2003-125344 disclosesan imaging device that shoots a high-resolution image (an image having aquality enough for viewing as a still image) during moving imageshooting, and a method of processing the shot image. Japanese PatentApplication Laid-Open No. 2003-125344 teaches that images are reproducedup to a predetermined resolution (high-resolution images up to the sameresolution as the moving image) by a progressive method during movingimage playback to enable viewing as a moving image, while thehigh-resolution image is extracted and transferred as a still image inthe case of a still image application.

Although such an imaging device capable of shooting two images at thesame time as described in Japanese Patent Application Laid-Open No.2014-048459 can be expected to improve convenience by presenting twoimages properly, there is no specific mention on a useful presentationmethod.

Further, Japanese Patent Application Laid-Open No. 2003-125344 does notpresent a preferred playback mode of switching between the moving imageand the still image at arbitrary times.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an imaging devicecapable of acquiring images suitable for viewing of both a moving imageand a still image, and a playback device capable of presenting theacquired images properly.

According to one aspect of the present invention, there is provided animaging element that acquires a first image based on signal chargegenerated during a first accumulation time, and a second image based onsignal charge generated during a second accumulation time relativelylonger than the first accumulation time and recorded in synch with thefirst image during a synchronization period including the firstaccumulation time, and a moving image file generating unit thatgenerates a moving image file including a first moving image based onthe first image, a second moving image based on the second image, andsynchronization information for synchronizing the first moving image andthe second moving image frame by frame.

According to another aspect of the present invention, there is provideda playback device including a playback unit that playbacks a movingimage file captured by an imaging device that acquires a first imagebased on signal charge generated during a first accumulation time, and asecond image based on signal charge generated during a secondaccumulation time relatively longer than the first accumulation time andrecorded in synch with the first image during a synchronization periodincluding the first accumulation time, wherein the playback unitincludes as modes of playbacking the moving image file a firstpresentation mode without any change in presented image with time, and asecond presentation mode to change the presented image with time,wherein a first moving image based on the first image is selected fromthe moving image file and presented in the first presentation mode, anda second moving image based on the second image is selected from themoving image file and presented in the second presentation mode.

According to still another aspect of the present invention, there isprovided a playback method of playbacking a moving image file shot withan imaging device that acquires a first image based on signal chargegenerated during a first accumulation time, and a second image based onsignal charge generated during a second accumulation time relativelylonger than the first accumulation time and recorded in synch with thefirst image during a synchronization period including the firstaccumulation time, the method including selecting and presenting a firstmoving image based on the first image from the moving image fileaccording to a playback instruction in a first presentation mode withoutany change in presented image with time, and selecting and presenting asecond moving image based on the second image from the moving image fileaccording to a playback instruction in a second presentation mode tochange the presented image with time.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B are external views of an imaging device according toa first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a schematic configuration of theimaging device according to the first embodiment of the presentinvention.

FIG. 3 is a block diagram illustrating a configuration example of animaging element of the imaging device according to the first embodimentof the present invention.

FIG. 4 is a cross-sectional view illustrating the internal structure ofthe imaging element in the imaging device according to the firstembodiment of the present invention.

FIG. 5 is a graph illustrating a relationship between the angle of alight beam incident on a pixel and output from photodiodes.

FIG. 6A and FIG. 6B are diagrams illustrating the relationship between aphotographing optical system and the imaging element in the imagingdevice according to the first embodiment of the present invention.

FIG. 7A, FIG. 7B, and FIG. 7C are schematic diagrams for describingimage signals output from the imaging element.

FIG. 8 is a circuit diagram illustrating a configuration example of eachpixel of the imaging element of the imaging device according to thefirst embodiment of the present invention.

FIG. 9 and FIG. 10 are planar layout diagrams illustrating the main partof each pixel of the imaging element of the imaging device according tothe first embodiment of the present invention.

FIG. 11 is a circuit diagram illustrating a configuration example ofreadout circuits of the imaging element of the imaging device accordingto the first embodiment of the present invention.

FIG. 12 is a timing chart illustrating a driving sequence of the imagingelement.

FIG. 13 is a graph illustrating temporal changes in signal charge inphotodiodes.

FIG. 14A, FIG. 14B, and FIG. 14C are potential diagrams of the pixeltaken along A-B line in FIG. 9.

FIG. 15 is a cross-sectional view illustrating the propagation of lightand the behavior of electric charges generated by the photoelectricconversion inside the imaging element.

FIG. 16 is a timing chart for describing an imaging sequence in theimaging device according to the first embodiment of the presentinvention.

FIG. 17 is a diagram illustrating an example of time code values addedto each frame of moving image data.

FIG. 18 is a diagram illustrating an example of the file structure of“picture A” and “picture B.”

FIG. 19 is a diagram for describing a shooting condition setting screenfor “picture A” and “picture B.”

FIG. 20 is a diagram illustrating a relationship between ISO sensitivityranges of “picture A” and “picture B.”

FIG. 21 is a program AE chart in a dual image mode of the imaging deviceaccording to the first embodiment of the present invention.

FIG. 22 is a chart for describing a shutter speed difference between“picture A” and “picture B” along an imaging sequence.

FIG. 23 is a diagram illustrating a state of a display unit during liveview display after the imaging device is powered up.

FIG. 24A and FIG. 24B are diagrams illustrating one frame among imageframes acquired by operating a switch ST and a switch MV.

FIG. 25 is a flowchart illustrating a series of processing proceduresteps including crosstalk correction.

FIG. 26 is a diagram for describing crosstalk correction processingperformed in a digital signal processing unit.

FIG. 27 is a graph illustrating a specific example of crosstalkcorrection functions.

FIG. 28 is a diagram illustrating an example of an image after beingsubjected to crosstalk correction.

FIG. 29 is a diagram illustrating a state of displaying “picture A” and“picture B” next to each other on a display unit.

FIG. 30 is a diagram for describing an image playback method accordingto the first embodiment of the present invention.

FIG. 31A, FIG. 31B, FIG. 31C, FIG. 31D, and FIG. 31E are diagrams fordescribing file formats for storing “picture A” and “picture B.”

FIG. 32 is a circuit diagram illustrating a configuration example ofpixels of an imaging element of the imaging device according to a thirdembodiment of the present invention.

FIG. 33 is a program AE chart in a dual image mode of the imaging deviceaccording to the third embodiment of the present invention.

FIG. 34 is a flowchart illustrating a method of driving the imagingdevice according to the third embodiment of the present invention.

FIG. 35 is a chart for describing a method of driving the imagingelement in a first moving image/still image shooting mode.

FIG. 36 is a timing chart illustrating a driving sequence of the imagingelement in the first moving image/still image shooting mode.

FIG. 37 is a chart for describing a method of driving the imagingelement in a second moving image/still image shooting mode.

FIG. 38 is a timing chart illustrating a driving sequence of the imagingelement in the second moving image/still image shooting mode.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

First Embodiment

An imaging device according to a first embodiment of the presentinvention will be described with reference to FIG. 1A to FIG. 30. In thepresent embodiment, description will be made by taking, as an example ofa preferred embodiment of the present invention, an imaging deviceincluding an imaging element, a photographing optical system, and thelike for imaging, and an image playback device. Note that the imageplayback device is not necessarily required to be part of the imagingdevice, and it may be configured in hardware different from that of theimaging element and the photographing optical system. Further, theoverall or part of the function of the image playback device may beincluded in the imaging element.

FIG. 1A and FIG. 1B are external views of a digital still motion cameraas an example of the imaging device according to the present embodiment.FIG. 1A illustrates a front view and FIG. 1B illustrates a back view.

An imaging device 100 according to the present embodiment includes ahousing 151, a photographing optical system 152 provided in a frontportion of the housing 151, and a switch ST 154 and a propeller 162provided on the top face of the housing 151. The imaging device 100 alsoincludes, on the back side of the housing 151, a display unit 153, aswitch MV 155, a shooting mode selecting lever 156, a menu button 157,up and down switches 158, 159, a dial 160, and a playback button 161.

The housing 151 is a case for housing various functional parts, such asthe imaging element, a shutter, and the like, which constitute theimaging device 100. The photographing optical system 152 is an opticalsystem for forming an optical image of an object. The display unit 153is configured to include a display for displaying photographicinformation and an image. A movable mechanism may be provided in thedisplay unit 153 to angle a screen as necessary. The display unit 153has a display brightness range capable of displaying an image having awide dynamic range without suppressing the brightness range of theimage. The switch ST 154 is a shutter button mainly used to shoot astill image. The switch MV 155 is a button used to start or stop movingimage shooting. The shooting mode selecting lever 156 is a selectorswitch for selecting a shooting mode. The menu button 157 is a button tomove to a function setting mode for setting the function of the imagingdevice 100. The up and down switches 158, 159 are buttons used to changevarious set values. The dial 160 is a dial for changing various setvalues. The playback button 161 is a button to move to a playback modefor playbacking, on the display unit 153, an image recorded on arecording medium housed in the imaging device 100. The propeller 162 isto make the imaging device 100 float in the air in order to take imagesfrom the air.

FIG. 2 is a block diagram illustrating a schematic configuration of theimaging device 100 according to the present embodiment. As illustratedin FIG. 2, the imaging device 100 includes an aperture 181, an aperturecontrol unit 182, an optical filter 183, an imaging element 184, analogfront ends 185, 186, digital signal processing units 187, 188, and atiming generation unit 189. The imaging device 100 also includes asystem control CPU 178, a switch input unit 179, an image memory 190,and a flight controller 200. Further, the imaging device 100 includes adisplay interface unit 191, a recording interface unit 192, a recordingmedium 193, a print interface unit 194, an external interface unit 196,and a radio interface unit 198.

The imaging element 184 is to convert an optical image of an objectformed through the photographing optical system 152 into an electricalimage signal. Though not particularly limited, the imaging element 184includes the number of pixels, the signal readout rate, the color gamut,and the dynamic range enough to meet a standard such as the UHDTV (UltraHigh Definition Television) standard. The aperture 181 is to adjust theamount of light passing through the photographing optical system 152.The aperture control unit 182 is a circuit or a processor configured tocontrol the aperture 181. The optical filter 183 is to limit thewavelength of light incident on the imaging element 184 and the spatialfrequency to be transmitted to the imaging element 184. Thephotographing optical system 152, the aperture 181, the optical filter183, and the imaging element 184 are disposed on an optical axis 180 ofthe photographing optical system 152.

The analog front ends 185, 186 are a circuit or a processor configuredto perform analog signal processing and analog-digital conversionprocessing of an image signal output from the imaging element 184. Eachof the analog front ends 185, 186 is, for example, composed of acorrelated double sampling (CDS) circuit for removing noise, anamplifier for adjusting signal gain, an A/D converter for converting ananalog signal to a digital signal, and the like. The digital signalprocessing units 187, 188 are to compress image data after makingvarious corrections to digital image data output from the analog frontends 185, 186. The corrections made by the digital signal processingunits 187, 188 include crosstalk correction to be described later. Thetiming generation unit 189 is a circuit or a processor configured tooutput various timing signals to the imaging element 184, the analogfront ends 185, 186, and the digital signal processing units 187, 188.The system control CPU 178 is a control unit for carrying out variousoperations and performing overall control of the imaging device 100. Theimage memory 190 is to temporarily store image data.

The display interface unit 191 is an interface between the systemcontrol CPU 178 and the display unit 153 to display a shot image in thedisplay unit 153. The recording medium 193 is a recording medium such asa semiconductor memory to record image data, additional data, and thelike, which may be equipped in the imaging device 100 or be removable.The recording interface unit 192 is an interface between the systemcontrol CPU 178 and the recording medium 193 to perform recording on therecording medium 193 or reading from the recording medium 193. Theexternal interface unit 196 is an interface between the system controlCPU 178 and an external device to communicate with the external devicesuch as an external computer 197. The print interface unit 194 is aninterface between the system control CPU 178 and a printer 195 to outputa shot image to the printer 195 such as a small ink-jet printer in orderto print out the shot image. The radio interface unit 198 is aninterface between the system control CPU 178 and a network 199 such asthe Internet to communicate with the network 199. The switch input unit179 includes plural switches to switch various modes, such as the switchST 154 and the switch MV 155. The flight controller 200 is a controllerto control the propeller 162 so as to fly the imaging device 100 inorder to do shooting from the air.

In an imaging device including an image playback device like the imagingdevice 100 described in the present embodiment, shot image data can beplaybacked using the display unit 153 or an external monitor. During theplayback of the image data, the image data and additional data are readout from the recording medium 193. The readout data are demodulated inthe digital signal processing units 187, 188 according to an instructionfrom the system control CPU 178 to be presented as an image in thedisplay unit 153 through the display interface unit 191. A user canoperate an operation part (the menu button 157, the up and down switches158, 159, the dial 160, and the like) provided on the back side of theimaging device 100 to control the operation during playback. The useroperations include the playback, stop, and pause of a moving image.

FIG. 3 is a block diagram illustrating a configuration example of theimaging element 184. As illustrated in FIG. 3, the imaging element 184includes a pixel array 302, a vertical scanning circuit 307, readoutcircuits 308A, 308B, and timing control circuits 309A, 309B.

In the pixel array 302, a plurality of pixels 303 are arranged in theshape of a matrix. Although the actual number of pixels 303 belonging tothe pixel array 302 is generally enormous, only 16 pixels 303 arrangedin a 4×4 matrix are illustrated here for the sake of simplifying thefigure. Each of the plurality of pixels 303 includes a pair of a pixelelement 303A and a pixel element 303B. In FIG. 3, the upper half area ofthe pixel 303 is the pixel element 303A, and the lower half area of thepixel 303 is the pixel element 303B. The pixel element 303A and thepixel element 303B generate signals by photoelectric conversion,respectively.

Signal output lines 304A, 304B extending in the column direction areprovided in each column of the pixel array 302, respectively. The signaloutput line 304A in each column is connected to the pixel elements 303Abelonging to the column. Signals from the pixel elements 303A are outputto the signal output line 304A. The signal output line 304B in eachcolumn is connected to the pixel elements 303B belonging to the column.Signals from the pixel elements 303B are output to the signal outputline 304B. Further, in each column of the pixel array 302, a powersource line 305 and a ground line 306 extending in the column directionare provided, respectively. The power source line 305 and the groundline 306 in each column are connected to the pixels 303 belonging to thecolumn. The power source line 305 and the ground line 306 may also besignal lines extending in the row direction.

The vertical scanning circuit 307 is arranged adjacent to the pixelarray 302 in the row direction. The vertical scanning circuit 307outputs predetermined control signals to the plurality of pixels 303 ofthe pixel array 302 in units of rows through unillustrated control linesarranged to extend in the row direction in order to control readoutcircuits in the pixels 303. In FIG. 3, a reset pulse φRESn and transferpulses φTXnA, φTXnB are illustrated as control signals (where n is aninteger corresponding to each row number).

The readout circuits 308A, 308B are arranged adjacent to the pixel array302 in the column direction to sandwich the pixel array 302therebetween. The readout circuit 308A is connected to the signal outputline 304A in each column. The readout circuit 308A selectively activatesthe signal output line 304A in each column sequentially to read signalsfrom the signal output line 304A in each column in a sequential orderand perform predetermined signal processing. Similarly, the readoutcircuit 308B is connected to the signal output line 304B in each column.The readout circuit 308B selectively activates the signal output line304B in each column sequentially to read signals from the signal outputline 304B in each column in a sequential order and perform predeterminedsignal processing. Each of the readout circuits 308A, 308B can include anoise cancellation circuit, an amplifier circuit, an analog/digitalconverter circuit, and a horizontal scanning circuit, respectively, tooutput signals after being subjected to the predetermined signalprocessing sequentially.

The timing control circuit 309A is connected to the vertical scanningcircuit 307 and the readout circuit 308A. The timing control circuit309A outputs a control signal to control the drive timing of thevertical scanning circuit 307 and the readout circuit 308A. The timingcontrol circuit 309B is connected to the vertical scanning circuit 307and the readout circuit 308B. The timing control circuit 309B outputs acontrol signal to control the drive timing of the vertical scanningcircuit 307 and the readout circuit 308B.

FIG. 4 is a cross-sectional view illustrating the internal structure ofeach pixel 303 of the imaging element 184. As illustrated in FIG. 4,each pixel 303 includes two photodiodes 310A, 310B, a light guide 255,and a color filter 256. The photodiode 310A forms part of the pixelelement 303A and the photodiode 310B forms part of the pixel element303B. The photodiodes 310A, 310B are provided in a silicon substrate251. The light guide 255 is provided in an insulating layer 254 providedover the silicon substrate 251. The insulating layer 254 is, forexample, made of silicon oxide, and the light guide 255 is made of amaterial having a higher refractive index than the insulating layer 254such as silicon nitride. Interconnection layers 252 are provided in theinsulating layer 254 between adjacent light guides 255. The color filter256 having predetermined spectral transmittance characteristics isprovided over the light guide 255. Note that FIG. 4 illustrates anexample in which color filters of adjacent two pixels 303 are colorfilters 256, 257 having spectral transmittance characteristics differentfrom each other.

The light guide 255 has the property of confining light therein due to arefractive index difference from the insulating layer 254. This enablesthe light guide 255 to guide light incident through the color filter 256to the photodiodes 310A, 310B. The photodiodes 310A, 310B are arrangedasymmetric with respect to the light guide 255, and a light fluxpropagating through the light guide 255 enters the photodiode 310A withrelatively high efficiency and enters the photodiode 310B withrelatively low efficiency. Further, the depth and inclined angle of thelight guide 255 can be adjusted to prevent nonuniformity in the incidentangle characteristics of incident light flux capable of being convertedphotoelectrically by the photodiodes 310A, 310B effectively.

FIG. 5 is a graph illustrating a relationship between the angle of alight beam incident on a pixel and output from photodiodes. In FIG. 5,the abscissa represents the angle of the light beam incident on thepixel, and the ordinate represents output from the photodiodes. In FIG.5, an output characteristic 261 from the photodiode 310A and an outputcharacteristic 262 from the photodiode 310B are illustrated.

As illustrated in FIG. 5, the characteristic 261 and the characteristic262 both exhibit gentle hill-like shapes symmetric about peaks, each ofwhich the incident angle of the light beam is zero. The peak intensityPB of the characteristic 262 is about ⅛ of the peak intensity PA of thecharacteristic 261. This means that the dependences of the photodiodes310A, 310B on the incident angle are both small and the light-receivingefficiency of the photodiode 310B is ⅛ of that of the photodiode 310A.In other words, when the sensitivity of the photodiode 310B is replacedby a set value of the ISO sensitivity, the sensitivity of the photodiode310B becomes lower by three steps than that of the photodiode 310A.

Next, a relationship between the photographing optical system 152 andthe imaging element 184 will be described more specifically withreference to FIG. 6A and FIG. 6B. FIG. 6A and FIG. 6B are diagrams fordescribing the relationship between the photographing optical system 152and the imaging element 184. FIG. 6A is a diagram when the photographingoptical system 152 is viewed from a direction along an optical axis 180.FIG. 6B is a diagram more specifically illustrating a section from thephotographing optical system 152 to the imaging element 184 in FIG. 2.

As illustrated in FIG. 6B, it is assumed that the imaging element 184includes a pixel 276 located in a central portion of the imaging areaand a pixel 277 located near the outer edge of the imaging area. In thiscase, the pixel 276 can receive a light flux from an area surrounded bya light beam 272 and a light beam 273. The pixel 277 can receive a lightflux from an area surrounded by a light beam 274 and a light beam 275.On this occasion, since a field lens 270 is arranged between the opticalfilter 183 and the photographing optical system 152, the light fluxreceived by the pixel 276 and the light flux received by the pixel 277are overlapped near the photographing optical system 152 as indicated byan area 271 in FIG. 6A. As a result, both of the pixels can receive thelight flux emitted from the photographing optical system 152 with highefficiency.

FIG. 7A to FIG. 7C are schematic diagrams for describing image signalsoutput from the imaging element. Suppose here that color filters havingpredetermined optical transmittance characteristics are arranged overthe pixel array 302 in the layout of a color filter array 281illustrated in FIG. 7A. FIG. 7A schematically illustrates the pixelarray 302 with the pixels 303 arranged in a matrix with six rows andeight columns, and the colors of color filters arranged on respectivepixels. In FIG. 7A, R denotes a red color filter, G1 and G2 denote greencolor filters, and B denotes a blue color filter, respectively. Theillustrated color filter array 281 is a color filter array, a so-calledBayer array, where color filters of respective colors are arrangedrepeatedly in each row like G1BG1B . . . , RG2RG2 . . . , G1BG1B . . . ,and so on.

Output data 282, 283 illustrated in FIG. 7B and FIG. 7C are obtainedfrom the pixel array 302 having such a color filter array 281. In FIG.7B, g1A and g2A represent output from pixel elements 303A of the pixels303 with the green color filters arranged thereon. bA represents outputfrom pixel elements 303A of the pixels 303 with the blue color filterarranged thereon. rA represents output from pixel elements 303A of thepixels 303 with the red color filter arranged thereon. In FIG. 7C, g1Band g2B represent output from pixel elements 303B of the pixels 303 withthe green color filters arranged thereon. bB represents output frompixel elements 303B of the pixels 303 with the blue color filterarranged thereon. rB represents output from pixel elements 303B of thepixels 303 with the red color filter arranged thereon.

As described with reference to FIG. 3, two types of output from thereadout circuits 308A, 308B can be obtained from the imaging element184. One type of output is the output data 282 illustrated in FIG. 7B,and the other type of output is the output data 283 illustrated in FIG.7C. The output data 282 are subjected to predetermined signal processingto generate an image signal “picture A.” The output data 283 aresubjected to predetermined signal processing to generate an image signal“picture B.” In the following description, the image signal based on theoutput data 282 is referred to as “picture A,” and the image signalbased on the output data 283 is referred to as “picture B.” Although the“picture A” and “picture B” are image signals after being subjected to apredetermined correction in a precise sense, image signals before orduring the correction may also be referred to as the “picture A” and“picture B” for the purpose of illustration.

FIG. 8 is a circuit diagram illustrating a configuration example of eachpixel 303. As mentioned above, the pixel 303 includes the pixel element303A and the pixel element 303B. The pixel element 303A includes thephotodiode 310A, a transfer transistor 311A, a floating diffusion region313A, a reset transistor 314A, and an amplifier transistor 315A. Thepixel element 303B includes the photodiode 310B, a transfer transistor311B, a floating diffusion region 313B, a reset transistor 314B, and anamplifier transistor 315B. Note that the photodiode 310A corresponds tothe photodiode 310A illustrated in FIG. 4, and the photodiode 310Bcorresponds to the photodiode 310B illustrated in FIG. 4.

The anode of the photodiode 310A is connected to the ground line 306,and the cathode of the photodiode 310A is connected to the source of thetransfer transistor 311A. The drain of the transfer transistor 311A isconnected to the source of the reset transistor 314A and the gate of theamplifier transistor 315A. A connection node of the drain of thetransfer transistor 311A, the source of the reset transistor 314A, andthe gate of the amplifier transistor 315A forms the floating diffusionregion 313A. The drain of the reset transistor 314A and the drain of theamplifier transistor 315A are connected to the power source line 305.The source of the amplifier transistor 315A that forms a pixel signaloutput part 316A is connected to the signal output line 304A.

Similarly, the anode of the photodiode 310B is connected to the groundline 306, and the cathode of the photodiode 310B is connected to thesource of the transfer transistor 311B. The drain of the transfertransistor 311B is connected to the source of the reset transistor 314Band the gate of the amplifier transistor 315B. A connection node of thedrain of the transfer transistor 311B, the source of the resettransistor 314B, and the gate of the amplifier transistor 315B forms thefloating diffusion region 313B. The drain of the reset transistor 314Band the drain of the amplifier transistor 315B are connected to thepower source line 305. The source of the amplifier transistor 315B thatforms a pixel signal output part 316B is connected to the signal outputline 304B.

The pixels 303 in each row are connected to a reset control line 319 andtransfer control lines 320A, 320B arranged in the row direction from thevertical scanning circuit 307. The reset control line 319 is connectedto the gate of the reset transistor 314A and the gate of the resettransistor 314B. The transfer control line 320A is connected to the gateof the transfer transistor 311A via a contact part 312A. The transfercontrol line 320B is connected to the gate of the transfer transistor311B via a contact part 312B. The reset control line 319 supplies thereset pulse φRESn, output from the vertical scanning circuit 307, to thegate of the reset transistor 314A and the gate of the reset transistor314B. The transfer control line 320A supplies the transfer pulse φTXnA,output from the vertical scanning circuit 307, to the gate of thetransfer transistor 311A. The transfer control line 320B supplies thetransfer pulse φTXnB, output from the vertical scanning circuit 307, tothe gate of the transfer transistor 311B. Note that n attached to thereset pulse φRESn, the transfer pulse φTXnA, and the transfer pulseφTXnB is an integer corresponding to the row number. In FIG. 8, n isreplaced by an integer corresponding to the row number.

The photodiode 310A is a first photoelectric conversion unit thatgenerates electric charge by photoelectric conversion, and thephotodiode 310B is a second photoelectric conversion unit that generateselectric charge by photoelectric conversion. The floating diffusionregions 313A, 313B are regions to accumulate electric charge. Thetransfer transistor 311A transfers, to the floating diffusion region313A, the electric charge generated by the photodiode 310A. The transfertransistor 311B transfers, to the floating diffusion region 313B, theelectric charge generated by the photodiode 310B.

When a high-level transfer pulse φTXnA is output from the verticalscanning circuit 307, the transfer transistor 311A is turned on toconnect the photodiode 310A and the floating diffusion region 313A.Similarly, when a high-level transfer pulse φTXnB is output from thevertical scanning circuit 307, the transfer transistor 311B is turned onto connect the photodiode 310B and the floating diffusion region 313B.When a high-level reset pulse φRESn is output from the vertical scanningcircuit 307, the reset transistors 314A, 314B are turned on to reset thephotodiodes 310A, 310B, and the floating diffusion regions 313A, 313B.

When a low-level transfer pulse φTXnA is output from the verticalscanning circuit 307, the transfer transistor 311A is turned off tocause the photodiode 310A to start accumulating signal charge generatedby the photoelectric conversion. After that, when the high-leveltransfer pulse φTXnA is output from the vertical scanning circuit 307,the transfer transistor 311A is turned on to transfer the signal chargeof the photodiode 310A to the floating diffusion region 313A. Then, theamplifier transistor 315A amplifies and outputs, to the signal outputline 304A, the voltage of the floating diffusion region 313A accordingto the amount of signal charge transferred from the photodiode 310A.

Similarly, when a low-level transfer pulse φTXnB is output from thevertical scanning circuit 307, the transfer transistor 311B is turnedoff to cause the photodiode 310B to start accumulating signal chargegenerated by the photoelectric conversion. After that, when thehigh-level transfer pulse φTXnB is output from the vertical scanningcircuit 307, the transfer transistor 311B is turned on to transfer thesignal charge of the photodiode 310B to the floating diffusion region313B. Then, the amplifier transistor 315B amplifies and outputs, to thesignal output line 304B, the voltage of the floating diffusion region313B according to the amount of signal charge transferred from thephotodiode 310B.

FIG. 9 and FIG. 10 are planar layout diagrams illustrating the main partof each pixel 303. Among the constituent elements of the pixel 303, thephotodiodes 310A, 310B, the transfer transistors 311A, 311B, and thefloating diffusion regions 313A, 313B are illustrated in FIG. 9. Theother circuit elements including the reset transistors 314A, 314B andthe amplifier transistors 315A, 315B are represented as a readoutcircuit part 321 in FIG. 9 to omit detailed illustration. Further, thesignal output lines 304A, 304B, and the power source line 305 arrangedin the vertical direction of the pixel 303 are omitted, and the contactparts of the reset control line 319, the power source line 305, and theground line 306 are omitted. In addition to the constituent elementsillustrated in FIG. 9, the light guide 255 described with reference toFIG. 4 is illustrated in FIG. 10. In the light guide 255, the shadedarea indicates a low refractive index area and the outlined blank areaindicates a high refractive index area, namely a light guiding area.

In FIG. 9 and FIG. 10, the contact part 312A is a contact part toconnect the transfer control line 320A and the gate of the transfertransistor 311A. The contact part 312B is a contact part to connect thetransfer control line 320B and the gate of the transfer transistor 311B.Each of the photodiodes 310A, 310B is a photoelectric conversion unitthat performs the photoelectric conversion, having a firstconductivity-type (e.g., p-type) semiconductor region and a secondconductivity-type (e.g., n-type) semiconductor region (n-type electronaccumulation region) to form a p-n junction with the firstconductivity-type semiconductor region. The second conductivity-typesemiconductor region of the photodiode 310A and the secondconductivity-type semiconductor region of the photodiode 310B areisolated by an isolation part 322.

The transfer transistors 311A, 311B, the contact parts 312A, 312B, andthe transfer control lines 320A, 320B are arranged line-symmetric orsubstantially line-symmetric to the isolation part 322 between thephotodiodes 310A, 310B, respectively. On the other hand, the light guide255 is arranged in a position deviated from the isolation part 322 asillustrated in FIG. 10. In other words, the photodiode 310A occupiesmost of the area of the bottom of the light guide 255, whereas thephotodiode 310B slightly overlaps the bottom of the light guide 255.Therefore, the light-receiving efficiency of the photodiode 310A isrelatively high, and the light-receiving efficiency of the photodiode310B is relatively low.

In the imaging element 184 according to the present embodiment, theratio of the light-receiving efficiency between the photodiodes 310A and310B is set to about 8:1, i.e., the difference in sensitivity is set toabout three steps. Then, two images are shot in the settings ofdifferent accumulation times to obtain nearly equal signal charge ineach pixel element. This can make both images be noiseless images havingexcellent S/N ratios, or can synthesize both images to obtain ahigh-definition HDR image. The details will be described later.

FIG. 11 is a circuit diagram illustrating a configuration example of thereadout circuits 308A, 308B of the imaging element 184. Assuming thereadout circuit 308A, “A” is suffixed to some constituent elements inFIG. 11. It should be understood that “B” will be suffixed tocorresponding constituent elements in the readout circuit 308B.

As illustrated in FIG. 11, the readout circuit 308A includes a clampcapacitor CO, a feedback capacitor Cf, an operational amplifier 406, areference voltage source 407, and a switch 423. One input terminal ofthe operational amplifier 406 is connected to the signal output line304A via the clamp capacitor CO. The feedback capacitor Cf and theswitch 423 are connected in parallel between the one input terminal andthe output terminal of the operational amplifier 406. The other inputterminal of the operational amplifier is connected to a referencevoltage source 407. The reference voltage source 407 supplies areference voltage Vref to the operational amplifier 406. The switch 423is a switch controlled by a signal PCOR to be turned on when the signalPCOR is at high level so as to short-circuit both ends of the feedbackcapacitor Cf.

The readout circuit 308A also includes switches 414, 415, 418, and 419,a capacitor CTSA, a capacitor CTNA, horizontal output lines 424, 425,and an output amplifier 421. The switches 414, 415 are switches thatcontrol the writing of pixel signals to the capacitors CTSA and CTNA.The switch 414 is a switch controlled by a signal PTSA to be turned onwhen the signal PTSA is at high level so as to connect the outputterminal of the operational amplifier 406 and the capacitor CTSA. Theswitch 415 is a switch controlled by a signal PTNA to be turned on whenthe signal PTNA is at high level so as to connect the output terminal ofthe operational amplifier 406 and the capacitor CTNA.

The switches 418, 419 are switches to control the output of pixelsignals, held in the capacitors CTSA and CTNA, to the output amplifier421. The switches 418, 419 are turned on in response to a control signalfrom a horizontal shift register. Thus, the signal written in thecapacitor CTSA is output to the output amplifier 421 via the switch 418and a horizontal output line 424. The signal written in the capacitorCTNA is output to the output amplifier 421 via the switch 419 and ahorizontal output line 425. The signal PCOR, the signal PTNA, and thesignal PTSA are signals supplied from the timing generation unit 189under the control of the system control CPU 178.

The readout circuit 308B also have a configuration equivalent to that ofthe readout circuit 308A. Note that a signal PTNB and a signal PTSB inthe following description are signals supplied from the timinggeneration unit 189 under the control of the system control CPU 178,having roles equivalent to the signal PTNA and the signal PTSA in thereadout circuit 308A.

Next, reset, accumulation, and readout operations in the imaging element184 will be sequentially described with reference to a timing chart ofFIG. 12 by taking, as an example, reading operation from pixels 303 inthe first row.

First, at time t1, the vertical scanning circuit 307 shifts the transferpulses φTX1A, TX1B output to the transfer control lines 320A, 320B fromthe low level to the high level. Thus, the transfer transistors 311A,311B are turned on. At this time, since the high-level reset pulse φRES1is output to the reset control line 319 from the vertical scanningcircuit 307, the reset transistors 314A, 314B are also in the on-state.Therefore, the photodiodes 310A, 310B are connected to the power sourceline 305 via the transfer transistors 311A, 311B and the resettransistors 314A, 314B to get into the reset state. On this occasion,the floating diffusion regions 313A, 313B are also in the reset state.

Then, at time t2, the vertical scanning circuit 307 shifts the transferpulse φTX1B from the high level to the low level. Thus, the transfertransistor 311B is turned off to cause the photodiode 310B to startaccumulating signal charge by the photoelectric conversion.

Then, at time t3, the vertical scanning circuit 307 shifts the transferpulse φTX1A from the high level to the low level. Thus, the transfertransistor 311A is turned off to cause the photodiode 310A to startaccumulating signal charge by the photoelectric conversion.

Then, at time t4, the vertical scanning circuit 307 shifts the resetpulse φRES1 from the high level to the low level. Thus, the resettransistors 314A, 314B are turned off to release the rest of thefloating diffusion regions 313A, 313B.

Accordingly, the potential of the floating diffusion region 313A is readout as a pixel signal of a reset signal level to the signal output line304A via the amplifier transistor 315A, and input to the readout circuit308A. Further, the potential of the floating diffusion region 313B isread out as a pixel signal of a reset signal level to the signal outputline 304B via the amplifier transistor 315B, and input to the readoutcircuit 308B.

At time t4, since the high-level signal PCOR is output from the timinggeneration unit 189 to the readout circuit 308A and the readout circuit308B, the switch 423 is in the on-state. Therefore, the pixel signal ofthe reset signal level from the pixel element 303A is input to thereadout circuit 308A in a state where the operational amplifier 406buffers the output of the reference voltage Vref. Though notillustrated, the pixel signal of the reset signal level from the pixelelement 303B is also input to the readout circuit 308B in the samemanner.

Then, at time t5, the signal PCOR output from the timing generation unit189 to the readout circuit 308A and the readout circuit 308B is changedfrom the high level to the low level to turn off the switch 423.

Then, at time t6, the signal PTNA output from the timing generation unit189 to the readout circuit 308A is changed from the low level to thehigh level to turn on the switch 415 so that the output of theoperational amplifier 406 at the time will be written to the capacitorCTNA. Similarly, the signal PTNB output from the timing generation unit189 to the readout circuit 308B is changed from the low level to thehigh level to turn on the switch 415 so that the output of theoperational amplifier 406 at the time will be written to the capacitorCTNB.

Then, at time t7, the signal PTNA output from the timing generation unit189 to the readout circuit 308A is changed from the high level to thelow level to turn off the switch 415 so as to complete the writing tothe capacitor CTNA. Similarly, the signal PTNB output from the timinggeneration unit 189 to the readout circuit 308B is changed from the highlevel to the low level to turn off the switch 415 so as to complete thewriting to the capacitor CTNB.

Then, at time t8, the vertical scanning circuit 307 changes the transferpulses φTX1A, φTX1B from the low level to the high level to turn on thetransfer transistors 311A, 311B. Thus, the signal charge accumulated inthe photodiode 310A is transferred to the floating diffusion region313A, and the signal charge accumulated in the photodiode 310B istransferred to the floating diffusion 31B.

Since the end timings of the accumulation periods of the photodiodes310A, 310B are synchronized by changing the transfer pulses φTX1A, φTX1Bto the high level at time t8 at the same time, readout is done at thesame time after both complete the accumulation. Therefore, a crosstalkcorrection such as to correct data on “picture B” using data on “pictureA” or to correct data on “picture A” using data on “picture B” can bemade with a very simple arithmetical operation.

Then, at time t9, the vertical scanning circuit 307 changes the transferpulses φTX1A, φTX1B from the high level to the low level to turn off thetransfer transistors 311A, 311B. Thus, the readout of the signal chargeaccumulated in the photodiode 310A into the floating diffusion region313A and the readout of the signal charge accumulated in the photodiode310B into the floating diffusion region 313B are completed.

Accordingly, the potential of the floating diffusion region 313A, whichis changed by the signal charge, is read out as a pixel signal of anoptical signal level to the signal output line 304A via the amplifiertransistor 315A, and input to the readout circuit 308A. Further, thepotential of the floating diffusion region 313B, which is changed by thesignal charge, is read out as a pixel signal of an optical signal levelto the signal output line 304B via the amplifier transistor 315B, andinput to the readout circuit 308B.

Then, in the readout circuit 308A, voltage which is subjected toinverted gain with respect to a voltage change at a capacitance ratiobetween the clamp capacitor CO and the feedback capacitor Cf is outputfrom the operational amplifier 406. Similarly, in the readout circuit308B, voltage which is subjected to inverted gain with respect to thevoltage change at the capacitance ratio between the clamp capacitor COand the feedback capacitor Cf is output from the operational amplifier406.

Then, at time t10, the signal PTSA output from the timing generationunit 189 to the readout circuit 308A is changed from the low level tothe high level to turn on the switch 414 so that the output of theoperational amplifier 406 at the time will be written to the capacitorCTSA. Similarly, the signal PTSB output from the timing generation unit189 to the readout circuit 308B is changed from the low level the highlevel to turn on the switch 414 so that the output of the operationalamplifier 406 at the time will be written to the capacitor CTSB.

Then, at time t11, the signal PTSA output from the timing generationunit 189 to the readout circuit 308A is changed from the high level tothe low level to turn off the switch 414 so as to complete the writingto the capacitor CTSA. Similarly, the signal PTSB output from the timinggeneration unit 189 to the readout circuit 308B is changed from the highlevel to the low level to turn off the switch 414 so as to complete thewriting to the capacitor CTSB.

Then, at time t12, the vertical scanning circuit 307 changes the resetpulse φRES1 from the low level to the high level to turn on the resettransistors 314A, 314B. Thus, the floating diffusion regions 313A, 313Bare connected to the power source line 305 via the reset transistors314A, 314B to get into the reset state.

FIG. 13 is a graph illustrating temporal changes in signal chargegenerated by photoelectric conversion and accumulated in the photodiodes310A, 310B. In FIG. 13, the abscissa of the graph represents time andthe ordinate represents the amount of signal charge. On the time axis,time t1 to time t12 illustrated in FIG. 12 are marked.

At time t2, when the transfer pulse φTX1B is changed to the low level toturn off the transfer transistor 311B so as to start the accumulation ofsignal charge in the photodiode 310B, the amount of signal charge heldin the photodiode 310B increases with time. The increase in signalcharge continues until the transfer pulse φTX1B is changed to the highlevel at time t8 to turn on the transfer transistor 311B so as totransfer the signal charge of the photodiode 310B to the floatingdiffusion region 313B.

Further, at time t3, the transfer pulse φTX1A is changed to the lowlevel to turn off the transfer transistor 311A so as to start theaccumulation of signal charge in the photodiode 310A. Thus, the amountof signal charge held in the photodiode 310A increases with time. Theincrease in signal charge continues until the transfer pulse φTX1A ischanged to the high level at time t8 to turn on the transfer transistor311A so as to transfer the signal charge of the photodiode 310A to thefloating diffusion region 313A.

At time t8, a signal charge amount LB held in the photodiode 310B and asignal charge amount LA held in the photodiode 310A become substantiallythe same level by cancelling out the difference in light-receivingefficiency with the difference in accumulation time.

In a period TM1 where the transfer pulse φTX1B and the transfer pulseφTX1A are both at the low level, crosstalk occurs between the photodiode310A and the photodiode 310B. The period TM1 takes a value shorterbetween the accumulation period of the photodiode 310A and theaccumulation time of the photodiode 310B. Since the crosstalk amount isapproximately proportional to the amount of signal charge, relativelymore crosstalk occurs in a period TM2 as the second half of the periodTM1, where the signal charge amount increases.

A crosstalk amount CTAB from the photodiode 310A to the photodiode 310Bis proportional to the area of a region 953 indicated by hatchingdiagonally right down. A crosstalk amount CTBA from the photodiode 310Bto the photodiode 310A is proportional to the area of a region 954indicated by hatching diagonally left down. If these constants ofproportion are defined by k and g, respectively, the crosstalk amountsCTAB and CTBA can be expressed as follows.

CTAB=k×(LA×TM1)/2  (1)

CTBA=g×(LA+LBS)×TM1/2  (2)

LBS is a signal charge amount of the photodiode 310B at time t3.Further, though not illustrated in FIG. 13, an approximation to LB=LBScan be achieved if a period from time t2 to time t3 is sufficientlyshorter than the period TM1. Therefore, Equation (2) can be modified asfollows.

CTBA=g×LB×TM1  (3)

Thus, it is found from Equation (1) and Equation (3) that the crosstalkamount CTAB is a function of the signal charge amount LA and a value(period TM1) shorter between the accumulation time of the photodiode310A and the accumulation time of the photodiode 310B. It is also foundthat the crosstalk amount CTBA is a function of the signal charge amountLB and a value (period TM1) shorter between the accumulation time of thephotodiode 310A and the accumulation time of the photodiode 310B.

FIG. 14A to FIG. 14C are potential diagrams of the pixel 303 taken alongA-B line in FIG. 9. FIG. 14A is a potential diagram at time ta in FIG.12, FIG. 14B is a potential diagram at time tb in FIG. 12, and FIG. 14Cis a potential diagram at time tc in FIG. 12.

As illustrated in FIG. 14A, the transfer transistors 311A, 311B are inthe off-state at time ta, and signal charges at signal accumulationlevels 323A, 323B are accumulated in the photodiodes 310A, 310B,respectively. As mentioned above, although the photodiode 310A and thephotodiode 310B are different in light-receiving efficiency, the signalaccumulation levels 323A, 323B are substantially the same level bycancelling out the difference in light-receiving efficiency with thedifference in accumulation time. Since this state lasts a relativelylong time, a phenomenon that the accumulated electric charge of thephotodiode 310A leaks into the adjacent photodiode 310B and a phenomenonthat the accumulated electric charge of the photodiode 310B leaks intothe adjacent photodiode 310A occur at a non-negligible level.

As illustrated in FIG. 14B, the transfer transistors 311A, 311B are inthe on-state at time tb, and the potential barriers of the transfertransistors 311A, 311B are low. Thus, the signal charge accumulated inthe photodiode 310A is transferred to the floating diffusion region313A, and the signal charge accumulated in the photodiode 310B istransferred to the floating diffusion region 313B. On this occasion,although the potential barrier of the isolation part 322 is also low,the potential barriers of the transfer transistors 311A, 311B aresufficiently low. Therefore, the phenomena that the accumulated electriccharges of the photodiodes 310A, 310B leak into the adjacent photodiodes310B, 310A through the isolation part 322 at this timing hardly occur.

As illustrated in FIG. 14C, the transfer transistors 311A, 311B are inthe off-state at time tc, and the potentials return to the state in FIG.14A.

FIG. 15 is a cross-sectional view illustrating the behavior of electriccharges generated by the propagation of light and photoelectricconversion inside the imaging element 184. In FIG. 15, an arrow 451indicates a light flux entering the pixel 303. The light flux 451 firstenters the color filter 256, where a predetermined wavelength componentis absorbed, passes through an interfacial passivation film (notillustrated) corresponding to the uppermost part of the insulating layer254, and enters the light guide 255. As described above with referenceto FIG. 5, orientation information of the light beam, i.e., pupilinformation is lost inside the light guide 255 by the behavior of lightwaves. The light flux 451 moves on the side of the silicon substrate 251while being confined in the light guide 255 due to the refractive indexdifference between the light guide 255 and the insulating layer 254, andreaches the bottom of the light guide 255. The bottom of the light guide255 lies adjacent to the silicon substrate 251, and the light fluxemitted from the light guide 255 enters the silicon substrate 251. Thephotodiode 310A and the photodiode 310B provided adjacent to each otherinside the silicon substrate 251 are arranged to be greatly eccentricfrom the light guide 255. Therefore, a light flux 452 as most of thelight flux emitted from the light guide 255 enters the photodiode 310A,and a light flux 453 as part of the rest of the light flux emitted fromthe light guide 255 enters the photodiode 310B. The incident photons areconverted to signal charges in the photodiodes 310A, 310B.

On this occasion, signal charges generated inside the silicon substrate251 of the imaging element 184 may leak into adjacent pixel elements bydiffusion. For example, signal charge 454 generated in the photodiode310A leaks into the photodiode 310B by diffusion. Further, signal charge455 generated in the photodiode 310B leaks into the photodiode 310A bydiffusion. This phenomenon has an adverse effect on the image, resultingin a blur in the image.

FIG. 16 is a timing chart for describing an imaging sequence in theimaging device according to the present embodiment. The term “time code”on the top of the chart indicates time after power activation, and“00:00:00:00” indicates “Hr:Min:Sec:Frame.”

Time t31 is the power activation time of the imaging device 100.

At time t32, the switch MV 155 as a moving image shooting button isoperated by a user to be turned on to start imaging of “picture B” andimaging of “picture A” are started in response thereto. In response tooperating the switch MV 155 as the button to shoot a moving image, imagedata on the “picture B” are written onto the recording medium 193 afterbeing subjected to predetermined signal processing.

The reason for imaging the “picture A” simultaneously with imaging the“picture B” is to active a crosstalk correction to be described later atall times. Since the transfer transistor 311A will be in the on-stateunless the transfer pulse φTX1A illustrated in FIG. 13 is at the lowlevel, the signal charge generated in the photodiode 310A is neveraccumulated. However, if only the period of operating the switch ST 154is targeted for the crosstalk correction, the “picture B” recorded atthe operating timing of the switch ST 154 will be subjected to delicatebrightness variation or hue variation due to the influence of acrosstalk correction error.

During a period of time t33 to time t34 and a period of time t35 to timet36, the switch ST 154 used to shoot a still image is operated.Therefore, during these periods, image data on the “picture A” are alsowritten onto the recording medium 193 after being subjected topredetermined signal processing. The image data on the “picture A” mayalso be written onto the recording medium 193 during the same period asthat of the image data on the “picture B” in addition to the period oftime t33 to time t34 and the period of time t35 to time t36.

In both of the “picture A” and the “picture B,” it is assumed that eachpiece of image data recorded on the recording medium 193 is a movingimage at the same frame rate, e.g., 60 fps, and the NTSC time code isadded. For example, the time code value added to each frame of themoving image data is as illustrated in FIG. 17.

FIG. 18 is a diagram illustrating an example of the file structure ofimage data on “picture A” and “picture B.” Although an example of MP4file is illustrated as the format of image data here, the format ofimage data is not limited to this. The MP4 file format is standardizedin ISO/IEC 14496-1/AMD6. All pieces of information are stored in astructure called a Box, and composed of multiplexed video and audio bitstreams (media data) and management information (meta data) on thesepieces of media data. The box type of each Box is represented by anidentifier made up of four letters, respectively. A file type Box 501(ftyp) is located at the top of the file as a Box to identify the file.In a media data Box 502 (mdat), the video and audio bit streams aremultiplexed and stored. In a movie Box 503 (moov), managementinformation used to play back a stored bit stream stored in the mediadata Box 502 is stored. A skip Box 504 (skip) is a Box to skip datastored in the skip Box 504 during playback.

In the skip Box 504, a clip name 508 of a clip including this image datafile, and a clip UMID (Unique Material Identifier) 509 (CLIP-UMID)assigned to the material are stored. Also stored in the skip Box 504 area time code value (time code head value) 510 of a clip head frame, and aserial number 511 of a recording medium on which the material file isrecorded. In FIG. 18, a free space 505, user data 506, and meta data 507are also contained in the skip Box 504. Since special data such as UMIDof the material file and the serial number of the recording medium arestored in the skip Box, such data have no impact during playback on ageneral-purpose viewer.

The same CLIP-UMID is set for respective MP4 files of the “picture A”and “picture B.” This enables a search for a file having the sameCLIP-UMID from one material file using the CLIP-UMID to associate bothfiles mechanically without any human confirmation work.

FIG. 19 is a diagram for describing a shooting condition setting screenfor “picture A” and “picture B.” It is assumed that the shooting modeselecting lever 156 is rotated 90 degrees clockwise, for example, fromthe position in FIG. 1B to enter a dual image mode capable of shootingtwo images at the same time. A By value 521 corresponding to thebrightness of an object at the time, F-number 522, and respective ISOsensitivities 523, 524 and shutter speeds 525, 526 of the “picture A”and the “picture B” are displayed on the display unit 153. Further,picture modes 527, 528 currently set for the “picture A” and the“picture B” are displayed, respectively. A picture mode to suit thepurpose of shooting can be selected from among plural options using theup and down switches 158, 159, and the dial 160.

As mentioned above, the difference in light-receiving efficiency betweenthe photodiode 310A and the photodiode 310B is set as three-stepdifference. Therefore, there is a three-step difference in ISOsensitivity range between the “picture A” and the “picture B.” Asillustrated in FIG. 20, the ISO sensitivity range of the “picture A” isfrom ISO 100 to ISO 102400, and the ISO sensitivity range of the“picture B” is from ISO 12 to ISO 12800.

FIG. 21 is a program AE (Automatic Exposure) chart in the dual imagemode. The abscissa indicates Tv value and corresponding shutter speed,and the ordinate indicates Av value and corresponding aperture value.Further, the diagonal direction is equivalent By line. The relationshipbetween the By value and the ISO sensitivity of the “picture A” isrepresented in a gain notation area 556, and the relationship betweenthe By value and the ISO sensitivity of the “picture B” is representedin a gain notation area 557. In FIG. 21, each By value is represented asa numeric value surrounded by a rectangle to distinguish from the otherparameters.

Referring to FIG. 21, it will be described how the shutter speed, theaperture value, and the ISO sensitivity vary according to variationsfrom high brightness to low brightness.

First, when By value is 13, the ISO sensitivity of the “picture A” isset to ISO 100. The equivalent By line of the “picture A” intersectswith a program chart 558 of the “picture A” at point 551, and it isdetermined from the point 551 that the shutter speed is 1/4000 secondand the aperture value is F11. On the other hand, the ISO sensitivity ofthe “picture B” is set to ISO 12. The equivalent By line of the “pictureB” intersects with a program chart 559 of the “picture B” at point 552,and it is determined from the point 552 that the shutter speed is 1/500second and the aperture value is F11.

When By value is 10, the ISO sensitivity of the “picture A” increases byone step and is set to ISO 200. The equivalent By line of the “pictureA” intersects with the program chart 558 of the “picture A” at point553, and it is determined from the point 553 that the shutter speed is1/1000 second and the aperture value is F11. On the other hand, the ISOsensitivity of the “picture B” is set to ISO 12. The equivalent By lineof the “picture B” intersects with the program chart 559 of the “pictureB” at point 560, and it is determined from the point 560 that theshutter speed is 1/60 second and the aperture value is F11.

When By value is 6, the ISO sensitivity of the “picture A” is set to ISO200. The equivalent By line of the “picture A” intersects with theprogram chart 558 of the “picture A” at point 554, and it is determinedfrom the point 554 that the shutter speed is 1/1000 second and theaperture value is F2.8. On the other hand, the ISO sensitivity of the“picture B” is set to ISO 12. The equivalent By line of the “picture B”intersects with the program chart 559 of the “picture B” at point 555,and it is determined from the point 555 that the shutter speed is 1/60second and the aperture value is F2.8.

When By value is 5, the ISO sensitivity of the “picture A” increases byone step and is set to ISO 400. The equivalent By line of the “pictureA” intersects with the program chart 558 of the “picture A” at the point554, and it is determined from the point 554 that the shutter speed is1/1000 second and the aperture value is F2.8. On the other hand, the ISOsensitivity of the “picture B” is set to ISO 25. The equivalent By lineof the “picture B” intersects with the program chart 559 of the “pictureB” at the point 555, and it is determined from the point 555 that theshutter speed is 1/60 second and the aperture value is F2.8.

After that, as the brightness is reduced, gain-up is performed toincrease the ISO sensitivity without changing the shutter speed and theaperture value of both of the “picture A” and the “picture B.”

The exposure operation illustrated in this program AE chart is soperformed that the “picture A” will keep a shutter speed of 1/1000second or faster over the entire brightness range written, and the“picture B” will keep a shutter speed of 1/60 second over most of thebrightness range. Thus, a high-definition moving image with lessjerkiness can be obtained in the “picture B” while achieving the stopmotion effect in the “picture A.”

FIG. 22 is a chart for describing a shutter speed difference between the“picture A” and the “picture B” along an imaging sequence. In FIG. 22,the abscissa is expressed in time to illustrate a V synchronizing signal481, accumulation periods 482, 483 of the “picture A,” and accumulationperiods 484, 485 of the “picture B,” where n denotes a frame number.

The accumulation period 482 is an accumulation period of a screen upperedge line of the “picture A,” and the accumulation period 483 is anaccumulation period of a screen lower edge line of the “picture A.”Since the imaging element 184 performs exposure operation with thefunction of a rolling electronic shutter, the accumulation is started atpredetermined time intervals sequentially from the screen upper edgeline toward the screen lower edge line, and the accumulation is finishedsequentially at the time intervals. When the accumulation is completed,the signal charge is read out sequentially from the imaging element 184,and input to the analog front end 185. A period from time t53 to timet54 is the accumulation period 482, and a period from time t55 to timet56 is the accumulation period 483.

Further, the accumulation period 484 is an accumulation period of ascreen upper edge line of the “picture B,” and the accumulation period485 is an accumulation period of a screen lower edge line of the“picture B.” Like in the “picture A,” the accumulation in the “pictureB” is also started at predetermined time intervals from the screen upperedge line toward the screen lower edge line, and the accumulation isfinished sequentially at the time intervals. When the accumulation iscompleted, the signal charge is read out sequentially from the imagingelement 184, and input to the analog front end 186. A period from timet51 to time t54 is the accumulation period 484, and a period from timet52 to time t56 is the accumulation period 485.

Although the two images of the “picture A” and the “picture B” are shotin different accumulation time settings, similar levels of signal chargeare obtained in the imaging element 184, rather than performing thegain-up on the “picture A.” Therefore, both the “picture A” and the“picture B” become noiseless images having excellent S/N ratios.

FIG. 23 is a diagram illustrating a state of the display unit 153 duringlive view display after the imaging element 184 is powered up. A sportsscene of a person 163 captured through the photographing optical system152 is displayed on the display unit 153. Further, since the shootingmode selecting lever 156 is placed in a position turned 90 degreesclockwise from the state in FIG. 1B, shutter speeds 491, 492 of the“picture A” and the “picture B,” and an F-number 493 in the dual imagemode are displayed.

FIG. 24A and FIG. 24B illustrate one frame among image frames acquiredby operating the switch ST 154 and the switch MV 155, respectively. FIG.24A is an image of the “picture A” shot with a shutter speed of 1/1000second and an aperture value of F4.0. FIG. 24B is an image of the“picture B” shot with a shutter speed of 1/60 second and an aperturevalue of F4.0. The image illustrated in FIG. 24B is blurred due to sucha slow shutter speed that the motion of the object does not stop.However, if this image is played back as a moving image at a frame rateof about 60 fps, this blur will work rather well, leading to a smoothhigh-definition image with less jerkiness. On the other hand, the stopmotion effect is supposed to be seen in the image illustrated in FIG.24A because the shutter speed is fast. However, as previously describedwith reference to FIG. 15, signal charge generated inside the siliconsubstrate leaks into adjacent pixel elements by diffusion to result in ablurred image as if the image illustrated in FIG. 24B is added. Thiscrosstalk phenomenon also occurs in the image illustrated in FIG. 24B,but it is barely noticeable because the image is originally blurred.

Therefore, in the imaging device according to the present embodiment, acrosstalk correction to be described below is applied to an image signaloutput from the imaging element 184 in order to obtain an original stopmotion effect by the fast shutter speed.

FIG. 25 is a flowchart illustrating a series of processing proceduresteps including the crosstalk correction. Processing from imaging torecording in the imaging device 100 according to the present embodimentis performed, for example, in step S151 to step S155 illustrated in FIG.25.

In step S151, the accumulation of signal charge and readout of thesignal charge to the photodiodes 310A, 310B are performed according tothe sequence described with reference to FIG. 12 in response to theoperation of the switch MV 155 at time t32 as described with referenceto FIG. 16.

In step S152, signals read out from the imaging element 184 are input tothe analog front ends 185, 186, in which analog signals are digitized.

In step S153, a correction (crosstalk correction) to reduce crosstalkcaused by the leakage of signal charge generated inside the siliconsubstrate into adjacent pixel elements is performed. The crosstalkcorrection is performed in the digital signal processing units 187, 188.In other words, the digital signal processing units 187, 188 function ascrosstalk correction units.

In step S154, development processing and compression processing asneeded are performed. In the development processing, a gamma correctionis performed as one of a series of processing steps. The gammacorrection is processing to apply a gamma function to an input lightamount distribution. As a result, the linearity of the output withrespect to the input light amount distribution is not kept, and thecrosstalk ratio also varies with the light amount at the time.Therefore, as illustrated in FIG. 25, it is desired to perform thecrosstalk correction in a stage prior to step S154. When the crosstalkcorrection is performed after the development, the crosstalk processingmay be changed depending on the magnitude of the light amount, or thecrosstalk correction may be performed after the image signals aresubjected to inverse gamma correction.

In step S155, images are recorded on the recording medium 193. Insteadof or in addition to recording on the recording medium 193, the imagesmay also be stored in a storage device on a network 199 through theradio interface 198.

FIG. 26 is a diagram for describing the crosstalk correction processingperformed in step S153 by the digital signal processing units 187, 188.Actual processing is performed as digital signal processing.

In the digital signal processing unit 187, a signal 471A after beingsubjected to A/D conversion processing is input to a crosstalk amountcorrecting part 473A, and further input to a crosstalk amount correctingpart 473B via a crosstalk amount calculating part 472A. Similarly, inthe digital signal processing unit 188, a signal 471B after beingsubjected to A/D conversion processing is input to the crosstalk amountcorrecting part 473B, and further input to the crosstalk amountcorrecting part 473A via a crosstalk amount calculating part 472B.

In the crosstalk amount correcting part 473A, a crosstalk correction isperformed on the signal 471A based on the signal 471A and the signal471B after being subjected to a predetermined calculation by a crosstalkcorrection function gij(n) in the crosstalk amount calculating part 472Bto obtain an output signal 474A. The output signal 474A is subjected todevelopment and/or compression processing as a subsequent processingstep in the digital signal processing unit 187.

In the crosstalk amount correcting part 473B, a crosstalk correction isperformed on the signal 471B based on the signal 471B and the signal471A after being subjected to a predetermined calculation by a crosstalkcorrection function fij(n) in the crosstalk amount calculating part 472Ato obtain an output signal 474B. The output signal 474B is subjected todevelopment and/or compression processing as a subsequent processingstep in the digital signal processing unit 188.

Since the crosstalk depends on the amount of generated signal charge,the crosstalk amount correcting parts 473A, 473B can perform crosstalkcorrections in a manner to correct an output signal of one pixel elementby a crosstalk amount corresponding to the amount of signal chargegenerated in the other pixel element. This can remove, from the outputsignal of the one pixel element, a crosstalk component from the otherpixel element, which is superimposed on the output signal.

Here, data at a pixel address ij of the n-th frame of “picture A” aredenoted as DATA_Aij(n), data at a pixel address ij of the n-th frame of“picture B” is denoted as DATA_Bij(n), and a correction coefficient isdenoted as α. Since the crosstalk depends on the input light amount,corrected data C_DATA_Aij(n) at a pixel address ij of the n-th frame of“picture A” can be expressed as Equation (4).

C_DATA_Aij(n)=DATA_Aij(n)−α×DATA_Bij(n)  (4)

When a crosstalk correction function fij(n) is

fij(n)=−α×DATA_Bij(n),

Equation (4) can be expressed as follows.

C_DATA_Aij(n)=DATA_Aij(n)+fij(n).

Similarly, corrected data C_DATA_Bij(n) at a pixel address ij of then-th frame of “picture B” can be expressed as Equation (5) with thecorrection coefficient denoted as β.

C_DATA_Bij(n)=DATA_Bij(n)−β×DATA_Aij(n)  (5)

When a crosstalk correction function gij(n) is

gij(n)=−β×DATA_Aij(n),

Equation (5) can be expressed as follows.

C_DATA_Bij(n)=DATA_Bij(n)+gij(n)  (6).

As mentioned above, although crosstalk also occurs in the “picture B,”since it is barely noticeable because the image is originally blurred,processing expressed in Equation (5) and Equation (6) may be omitted. Ifthe crosstalk correction is performed on an image with a relativelyshort accumulation time without performing the crosstalk correction onan image with a relatively long accumulation time, the calculation loadcan be reduced.

FIG. 27 is a graph illustrating a specific example of the crosstalkcorrection functions fij(n), gij(n). In FIG. 27, the abscissa indicatesthe size of input data, and the ordinate indicates crosstalk correctionamount to be corrected. Both of the crosstalk correction functionsfij(n), gij (n) are functions to obtain crosstalk correction amountsproportional to the input data, respectively. Although both aredifferent depending on the pixel structure in a precise sense, thecorrection coefficient α and the correction coefficient β are nearlyequal numeric values. However, the degree of leakage of signal charge,generated inside the silicon substrate depending on the incident angleof light on each pixel element, into adjacent pixel elements bydiffusion is different. Therefore, the more the aperture 181 is openedto increase the F-number, the larger the crosstalk and hence the largerthe absolute value of the crosstalk correction amount. On the otherhand, the more the aperture 181 is narrowed to decrease the F-number,the smaller the crosstalk and hence the smaller the absolute value ofthe crosstalk correction amount. In FIG. 27, a characteristic 591 is acrosstalk correction function at F2.8, a characteristic 592 is acrosstalk correction function at F5.6, and a characteristic 593 is acrosstalk correction function at F11. The gradients of thecharacteristic 591, the characteristic 592, and the characteristic 593become smaller in this order. Note that the F-number of thephotographing optical system 152 can be continuously changed. Therefore,if the correction coefficient α and the correction coefficient β are setas an F-number function, more precise crosstalk corrections can beachieved.

Further, as previously described with reference to FIG. 13, thecorrection coefficient α and the correction coefficient β can be set asa function of the accumulation time of a photodiode for “picture A” setto be relatively short.

The crosstalk correction amount can also be changed depending on theimage height to achieve further more accurate crosstalk corrections.Since crosstalk increases when light enters the light guide 255obliquely, distance ZK from the optical axis 180 to each pixel may becalculated based on the pixel address ij to apply a crosstalk correctionso as to increase the absolute value in proportion to the distance ZK.Further, since the change in incident angle of light on the light guide255 depends also on distance HK between an exit pupil of thephotographing optical system 152 and the imaging element 184, thecrosstalk correction function can be set as a function of the distanceHK to perform more precise corrections.

FIG. 28 is an image of “picture A” after the image of “picture A” (FIG.24A) shot with a shutter speed of 1/1000 second and an aperture value ofF4.0 is subjected to a crosstalk correction. In the image of FIG. 24A,signal charge generated inside the silicon substrate leaks into adjacentpixel elements by diffusion to result in a blurred image as if the imageillustrated in FIG. 24B were added. On the other hand, in the image ofFIG. 28, the original stop motion effect by the fast shutter speed isachieved. On the display unit 153 of a digital still motion camera, forexample, it is desired to be able to display both “picture A” 496 and“picture B” 497 side by side or up and down as illustrated in FIG. 29when the playback button 161 is operated. Thus, the images can becompared to check on the level of the stop motion effect. Thisprocessing may also be performed in such a manner that image data aresupplied to a system or an apparatus through a network to cause thesystem or a computer of the apparatus to read and execute a program.

FIG. 30 is a diagram for describing a playback method in an imageplayback device for data files including “picture A” and “picture B”stored in a storage. As the image playback device, a tablet terminal, apersonal computer, a TV monitor, or the like can be used in addition tothe image playback device included in the imaging device 100 describedin the present embodiment. Components (such as a CPU, a demodulationunit, and a display unit) are provided in the device, such as the tabletterminal, the personal computer, or the TV monitor, to play back amoving image file like an MP4 file so as to serve as the image playbackdevice. In the imaging device 100 of the present embodiment, thefunction as the image playback unit is implemented mainly by the systemcontrol CPU 178.

It is assumed here that data files of “picture A” and “picture B” arestored in a storage on a network. In FIG. 30, a frame group 581 is aframe group of “picture A” stored in an MP4 file, and a frame group 571is a frame group of “picture B” stored in another MP4 file. The sameCLIP-UMID is set for these MP4 files to associate the MP4 files at thetime of shooting.

When the playback of a moving image is started, frames are played backsequentially from a head frame 572 of the frame group 571 of “picture B”at a predetermined frame rate. Since the “picture B” is shot in such asetting that the shutter speed is not excessively fast ( 1/60 second inthis example), the image playbacked is a high-definition image with lessjerkiness. In this specification, a playback mode for a moving imagefile when the moving image is playbacked at a frame rate higher than theframe rate at the time of shooting may be represented as a presentationmode to change presented images with time.

Suppose here that a user gives an instruction to change the playbackmode while a moving image of “picture B” is being presented. Forexample, when the user pauses the playback at the time where theplayback progresses up to a frame 573, a frame 582 with the same timecode is automatically retrieved from the data file of the “picture A”associated with the “picture B,” and the frame 582 is displayed. The“picture A” is shot with a fast shutter speed ( 1/1000 second in thisexample) at which the stop motion effect can be easily obtained, i.e.,the “picture A” is a powerful image obtained by shooting a moment of thesports scene. Although the two images of the “picture A” and the“picture B” are shot in different accumulation time settings, similarlevels of signal charge are obtained in the imaging element 184, ratherthan performing the gain-up on the “picture A.” Therefore, both the“picture A” and the “picture B” become noiseless images having excellentS/N ratios.

Here, when printing is instructed, data on the frame 582 of the “pictureA” are output to the printer 195 through the print interface 194. Thus,the print also become powerful one having the stop motion effect thatreflects the “picture A.”

When the user releases the pause, the procedure automatically returns tothe frame group 571 of the “picture B” to resume playback from a frame574. At this time, the image to be played back is a high-definitionimage with less jerkiness.

In the example of FIG. 30, although the frame presentation is changed tothe “picture A” when the playback of the “picture B” is paused, theframe presentation may be changed to the “picture A” when frame-by-frameplayback of the “picture B” is performed. In this specification, theplayback mode for a moving image file when playback is paused orframe-by-frame playback is performed may also be referred to as thepresentation mode without any change in presented image with time.Further, the frame presentation may be changed to the “picture A” whenthe playback is put in a mode to reduce the frame rate to a certain rateor slower so as to perform playback while checking the imagecontinuously frame by frame. In other words, it is convenient to presentthe “picture A” regardless of the present or absence of an instructionto switch to the presentation mode when the speed of frame-by-frameadvance is sufficiently slower than the normal playback frame rate (theframe rate at the time of shooting).

The above difference in effect on the image played back is considered tobe caused by the difference in image presentation method between thepresentation to change the presented image with time and thepresentation (including the frame-by-frame playback) without any changein presented image with time. In other words, the presentation methodvaries according to a presentation condition as to which of conflictingdemands is important, a demand for an image with less jerkiness or ademand for an image having a high stop motion effect.

In the present embodiment, in view of the image acquisition feature ofthe imaging device, an image based on signals from pixels whoseaccumulation times are relatively long is presented in the presentation(moving image presentation) to change the presented image with time. Onthe other hand, an image based on signals from pixels whose accumulationtimes are relatively short is presented in the presentation (still imagepresentation) without any change in presented image with time. Thus,images according to the conflicting demands, i.e., the image with lessjerkiness and the image having a high stop motion effect, can beprovided. This effect is very beneficial.

The image presentation method illustrated in the present embodiment canbe used to provide images suitable for viewing of both of movingimage/still image when two or more images are shot at the same time andviewed using a single imaging element.

Thus, according to the present embodiment, images suitable for viewingof both of a moving image and a still image can be acquired and playedback.

Second Embodiment

An imaging device according to a second embodiment of the presentinvention will be described with reference to FIG. 31A to FIG. 31E. Thesame constituent elements as those of the imaging device according tothe first embodiment illustrated in FIG. 1A to FIG. 30 are given thesame reference numerals to omit or simplify the description.

In the first embodiment, the method of generating two or more movingimage files in consideration of compatibility with conventional fileformats and automatically associating these moving image files isillustrated.

In the present embodiment, an example of another preferred file format,and association between “picture A” and “picture B” in this example willbe described. Note that the configuration of the imaging device used toobtain “picture A” and “picture B” is the same as that of the firstembodiment.

FIG. 31A is a schematic view illustrating the method described in thefirst embodiment. FIG. 31B is a schematic view illustrating a method tobe described in the present embodiment. FIG. 31C, FIG. 31D, and FIG. 31Eare diagrams for specifically describing image storing methods in thepresent embodiment.

In the method of the first embodiment, as illustrated in FIG. 31A, thesystem control CPU 178 separately generates a file 6001 as a movingimage file of “picture A” and a file 6002 as a moving image file of“picture B.” The system control CPU 178 has the function as a movingimage file generating unit. The file 6001 and the file 6002 areassociated using CLIP-UMID as described in the first embodiment. Inother words, the file 6001 contains a moving image of “picture A,” andsynchronization information for synchronizing the moving image of“picture A” and a moving image of “picture B” frame by frame. The file6002 contains the moving image of “picture B,” and synchronizationinformation for synchronizing the moving image of “picture A” and themoving image of “picture B” frame by frame.

In contrast, in the method of the present embodiment, the system controlCPU 178 generates one file 6003 from moving image data on “picture A”and moving image data on “picture B” as illustrated in FIG. 31B.Specific examples of storage methods into the file 6003 are illustratedin FIG. 31C to FIG. 31E.

The method illustrated in FIG. 31C is an example of using a stereo imageformat, so-called side-by-side. Since no parallax between “picture A”and “picture B” obtained by the imaging device of the first embodimentas illustrated in FIG. 5, a three-dimensional image is not obtained evenusing the stereo image format. Information is just stored using thestereo image format.

In the case of a stereo image, there is proposed a method (side-by-side)as one of methods for recoding an image presented to the right eye andan image presented to the left eye to store these images as one imagewith the images set laterally side-by-side. In the example of FIG. 31C,this method is used to store an image of “picture A” and an image of“picture B” as one image data with the images arranged as illustrated.The image of “picture A” and the image of “picture B” stored as oneimage are acquired in sync with a synchronization period. When focusingon a specific frame 6004 at this time, data on the frame 6004 is animage double in size in the lateral direction and composed of an image6005 of “picture A” and an image 6006 of “picture B” that lie next toeach other. In other words, the file 6003 is a moving image file inwhich each frame contains a frame image of a moving image of “pictureA,” and a frame image of a moving image of “picture B” acquired in syncwith that of “picture A.”

When the playback of the moving image is started, frame images areplayed back sequentially at a set frame rate from the head frame of aframe group of “picture B.” In other words, in a playback device, onlyan image to be present to one eye in the stereo image format iscontinuously presented. In the side-by-side method, an areacorresponding to the “picture B” can be clipped and presented. Since the“picture B” is shot in such a setting that the shutter speed will not beexcessively fast ( 1/60 second in this example), the image played backis a high-definition image with less jerkiness.

For example, when the user pauses the playback at the time where theplayback progresses up to the frame 6004, the image 6005 of “picture A”corresponding to the image 6006 of “picture B” is automaticallydisplayed. In other words, the image is switched to an image to bepresented to the other eye in the stereo image format. The “picture A”is shot with a fast shutter speed ( 1/1000 second in this example) atwhich the stop motion effect can be easily obtained, i.e., the “pictureA” is a powerful image obtained by shooting a moment of the sportsscene. Although the two images of the “picture A” and the “picture B”are shot in different accumulation time settings, similar levels ofsignal charge are obtained in the imaging element 184, rather thanperforming the gain-up on the “picture A.” Therefore, both the “pictureA” and the “picture B” become noiseless images having excellent S/Nratios.

The method illustrated in FIG. 31D is an example of using another stereoimage format suitable for a so-called liquid-crystal shutter typeplayback device. In the playback device using a liquid-crystal shutter,images are presented by switching an image presented to the right eyeand an image presented to the left eye in a time-division manner. Usingthis format, the “picture A” and the “picture B” can be stored in onefile. For example, when the frame rate of an image at the time ofshooting is designated as 60 fps, the “picture A” and the “picture B”are alternately stored as frames of a moving image at 120 fps, which istwice the frame rate. For example, a pair of an image of “picture A” andan image of “picture B” acquired in sync with each other are stored asdata on frames 6007 and 6008. Then, a pair of an image of “picture A”and an image of “picture B” acquired in sync with each other at the nexttiming are stored as data on frames 6009 and 6010. When focusing only ondata of either the “picture A” or the “picture B,” the data thus storedare data on a moving image at the same frame rate of 60 fps as that ofthe shooting time. In other words, the file 6003 is a moving image filerecorded to present frames of a moving image of “picture A” and framesof a moving image of “picture B” alternately. Then, the frames of themoving image of “picture A” and the frames of the moving image of“picture B” synchronized with each other are continuously recorded inthis moving image file.

When the playback of the moving image is started, frame images areplayed back sequentially at a set frame rate from the head frame of aframe group of “picture B.” In other words, in the playback device, onlyan image to be present to one eye in the stereo image format iscontinuously presented. In the example of FIG. 31D, every other framecan be played back to present only the “picture B.”

For example, when the user pauses the playback at the time where theplayback progresses up to the frame 6008, an image of the frame 6007 of“picture A” corresponding to the image of the frame 6008 of “picture B”is displayed. Thus, images suitable for viewing of both of movingimage/still image can be provided.

The method illustrated in FIG. 31E is an example of using a format tostore, in one file, two or more moving images as a multitrack movingimage. The format of FIG. 31E is a format capable of storing anauxiliary image, a parallax image, and the like as two or more tracks.Here, “picture B” is recorded as a main image 6012 of track 1, “pictureA” is recorded as an auxiliary image 6011 of track 2, and information onthe imaging device and the like is stored in a meta data recording part.Moving images on the two or more tracks correspond to one time code. Inother words, a file 6003 is a moving image file including a first movingimage track containing a moving image of “picture A” and synchronizationinformation, and a second moving image track containing a moving imageof “picture B” and synchronization information.

When the playback of the moving image is started, frame images areplayed back sequentially at a set frame rate from the head frame of aframe group of “picture B.” In other words, in the playback device, theimage of track is presented. When the user pauses the playback, an imageof track 2 corresponding to the same time code can be presented. Thus,images suitable for viewing of both of moving image/still image can beprovided.

Third Embodiment

An imaging device according to a third embodiment of the presentinvention will be described with reference to FIG. 32 to FIG. 38. Thesame constituent elements as those of the imaging devices according tothe first and second embodiments illustrated in FIG. 1A to FIG. 31E aregiven the same reference numerals to omit or simplify the description.

In the first and second embodiments, the two photodiodes 310A, 310Bdifferent in light-receiving efficiency (sensitivity) are used dependingon the accumulation time to enable moving image shooting suitable forvarious shooting scenes. In the present embodiment, an example ofcontrolling the accumulation time of one photodiode to achieve the sameeffect as that of the first and second embodiments will be described.

The imaging device according to the present embodiment is the same asthe imaging device according to the first embodiment except that thecircuit configuration of the pixels 303 of the imaging element 184 isdifferent.

FIG. 32 is a circuit diagram illustrating a circuit configuration of thepixels 303 of the imaging element 184 of the imaging device according tothe present embodiment. FIG. 32 illustrates a pixel 303 in the firstcolumn and the first row and a pixel 303 in the first column and them-th row among the plurality of pixels 303 that constitute the pixelarray 302. As illustrated in FIG. 32, each pixel 303 includes aphotodiode 600, transfer transistors 601A, 601B, 602A, 602B, and 603, areset transistor 604, an amplifier transistor 605, and a selecttransistor 606.

The anode of the photodiode 600 is connected to the ground line. Thecathode of the photodiode 600 is connected to the source of the transfertransistor 601A, the source of the transfer transistor 601B, and thesource of the transfer transistor 603, respectively. The drain of thetransfer transistor 601A is connected to the source of the transfertransistor 602A. A connection node between the drain of the transfertransistor 601A and the source of the transfer transistor 602A forms asignal holding unit 607A. The drain of the transfer transistor 601B isconnected to the source of the transfer transistor 602B. A connectionnode between the drain of the transfer transistor 601B and the source ofthe transfer transistor 602B forms a signal holding unit 607B.

The drain of the transfer transistor 602A and the drain of the transfertransistor 602B are connected to the source of the reset transistor 604and the gate of the amplifier transistor 605. A connection node of thedrain of the transfer transistor 602A, the drain of the transfertransistor 602B, the source of the reset transistor 604, and the gate ofthe amplifier transistor 605 forms a floating diffusion region 608. Thesource of the amplifier transistor 605 is connected to the drain of theselect transistor 606. The drain of the reset transistor 604 and thedrain of the amplifier transistor 605 are connected to a power sourceline 620. The drain of the transfer transistor 603 is connected to apower source line 621. The source of the select transistor 606 isconnected to a signal output line 623.

Thus, each pixel 303 of the imaging element 184 of the imaging deviceaccording to the present embodiment includes two signal holding units607A, 607B for one photodiode 600. Since the basic structure of a CMOStype imaging element 184 having signal holding units is disclosed, forexample, in Japanese Patent Application Laid-Open No. 2013-172210 by theapplicant of the present application, detailed description thereof willbe omitted here.

The plurality of pixels 303 of the pixel array 302 are connected inunits of rows to control lines arranged in the row direction from thevertical scanning circuit 307. The control lines in each row include aplurality of control lines connected to the gates of the transfertransistors 601A, 602A, 601B, 602B, and 603, the reset transistor 604,and the select transistor 606, respectively. The transfer transistor601A is controlled by a transfer pulse φTX1A, and the transfertransistor 602A is controlled by a transfer pulse φTX2A. The transfertransistor 601B is controlled by a transfer pulse φTX1B, and thetransfer transistor 602B is controlled by a transfer pulse φTX2B. Thereset transistor 604 is controlled by a reset pulse φRES, and the selecttransistor 606 is controlled by a select pulse φSEL. The transfertransistor 603 is controlled by a transfer pulse φTX3. Each controlpulse is sent out from the vertical scanning circuit 307. Eachtransistor is on-state when the control pulse is at the high level, andoff-state when the control pulse is at the low level.

The imaging element 184 that forms part of the imaging device of thepresent embodiment includes the two signal holding units 607A, 607B forone photodiode 600. This enables a first moving image having a stopmotion effect and a second moving image with less jerkiness to be shotat the same time. Therefore, two images different in accumulation periodcan be read out without reducing the S/N ratios.

The shooting conditions for the first moving image (corresponding to“picture A”) and the second moving image (corresponding to “picture B”)in the imaging device may be set in the same way as those in the firstand second embodiments.

FIG. 33 is a program AE chart in the dual image mode. The abscissaindicates Tv value and corresponding shutter speed, and the ordinateindicates Av value and corresponding aperture value. Further, thediagonal direction is equivalent By line. The relationship between theBy value and the ISO sensitivity of the first moving image (“picture A”)is represented in a gain notation area 556, and the relationship betweenthe By value and the ISO sensitivity of the second moving image(“picture B”) is represented in a gain notation area 557. In FIG. 33,each By value is represented as a numeric value surrounded by arectangle to distinguish from the other parameters.

Referring to FIG. 33, it will be described how the shutter speed, theaperture value, and the ISO sensitivity vary according to variationsfrom high brightness to low brightness. Since the imaging device of thepresent embodiment is used to shoot the first moving image and thesecond moving image at the same time, the same aperture value is set forthe same object brightness in the program AE chart.

First, when By value is 14, the ISO sensitivity of the first movingimage is set to ISO 100. The equivalent By line of the first movingimage intersects with a program chart 558 of the first moving image atpoint 551, and it is determined from the point 551 that the shutterspeed is 1/4000 second and the aperture value is F11. On the other hand,the ISO sensitivity of the second moving image is set to ISO 1. Theequivalent By line of the second moving image intersects with a programchart 559 of the second moving image at point 552, and it is determinedfrom the point 552 that the shutter speed is 1/60 second and theaperture value is F11.

When By value is 11, the ISO sensitivity of the first moving imageincreases by one step and is set to ISO 200. The equivalent By line ofthe first moving image intersects with the program chart 558 of thefirst moving image at point 553, and it is determined from the point 553that the shutter speed is 1/1000 second and the aperture value is F11.On the other hand, the ISO sensitivity of the second moving image is setto ISO 12. The equivalent By line of the second moving image intersectswith the program chart 559 of the second moving image at the point 552,and it is determined from the point 552 that the shutter speed is 1/60second and the aperture value is F11.

When By value is 7, the ISO sensitivity of the first moving image is setto ISO 200. The equivalent By line of the first moving image intersectswith the program chart 558 of the first moving image at point 554, andit is determined from the point 554 that the shutter speed is 1/1000second and the aperture value is F2.8. On the other hand, the ISOsensitivity of the second moving image is set to ISO 12. The equivalentBy line of the second moving image intersects with the program chart 559of the second moving image at point 555, and it is determined from thepoint 555 that the shutter speed is 1/60 second and the aperture valueis F2.8.

When By value is 6, the ISO sensitivity of the first moving imageincreases by one step and is set to ISO 400. The equivalent By line ofthe first moving image intersects with the program chart 558 of thefirst moving image at the point 554, and it is determined from the point554 that the shutter speed is 1/1000 second and the aperture value isF2.8. On the other hand, the ISO sensitivity of the second moving imageis set to ISO 25. The equivalent By line of the second moving imageintersects with the program chart 559 of the second moving image at thepoint 555, and it is determined from the point 555 that the shutterspeed is 1/60 second and the aperture value is F2.8.

After that, as the brightness is reduced, gain-up is performed toincrease the ISO sensitivity without changing the shutter speed and theaperture value of both of the first moving image and the second movingimage.

The exposure operation illustrated in this program AE chart is soperformed that the first moving image will keep a shutter speed of1/1000 second or faster over the entire brightness range written, andthe second moving image will keep a shutter speed of 1/60 second overthe entire brightness range. Thus, a high-definition moving image withless jerkiness can be obtained in the second moving image whileachieving the stop motion effect in the first moving image.

In the meantime, the first moving image and the second moving image shotwith the same aperture value at the same time are controlled to bedifferent in ISO sensitivity from each other. However, when exposurecontrol is performed to make the exposure of the first moving imageproper, the signal of the second moving image is so saturated that theISO sensitivity cannot be controlled. Therefore, in the imaging deviceaccording to the present embodiment, a short accumulation period isperformed Np times (where Np is an integer of 2 or more (Np>1)) at equalintervals while the shutter speed is 1/60 second corresponding to theframe rate of the second moving image. Then, the charge accumulatedbetween respective accumulation periods performed Np times is added upto generate an image to make the ISO sensitivity virtually low.

In the present embodiment, a period corresponding to the shutter speedof 1/60 second of the second moving image is set as a period duringwhich accumulation in the short accumulation period performed Np timesfor the second moving image is performed. Further, a periodcorresponding to the shutter speed of 1/1000 second of the first movingimage is set as an accumulation period for the first moving image (i.e.,the accumulation time for the first moving image is 1/1000 second).Then, the short accumulation period for the second moving image is socontrolled that the total accumulation time for the second moving imagewill become equal to the accumulation time for the first moving image.

In other words, the total accumulation time for the second moving imagegenerated by adding up charge accumulated during the short accumulationperiod performed Np times during the period corresponding to the shutterspeed of the second moving image is controlled to become equal to theaccumulation time for the first moving image. Further, each of theaccumulation times of Np times of accumulation periods for one secondmoving image is so controlled that the ISO sensitivity of the secondmoving image will become equal to the ISO sensitivity of a first movingimage shot during the shooting period of the second moving image.

As an example, suppose that when the brightness is Bv7, charge isaccumulated and added up 16 times during the period corresponding to theshutter speed of 1/60 second to generate the second moving image. Inthis case, each of the accumulation times of Np times of accumulationperiods for generation of the second moving image is set to 1/16000second to make the ISO sensitivity of the second moving image equivalentto the ISO sensitivity (ISO 200) of the first moving image.

FIG. 34 is a flowchart of shooting operation in the dual image mode toshoot the first moving image and the second moving image at the sametime. Since the first moving image is suitable for viewing of a stillimage having the stop motion effect, the first moving image may bereferred to as “still image” and the second moving image may be referredto as “moving image” in the following description to distinguish betweenthe first moving image and the second moving image. Further, theshooting mode in the present embodiment may be called a “movingimage/still image shooting mode” for descriptive purposes.

When the first moving image and the second moving image are shot at thesame time, the imaging device of the present embodiment can performshooting in either of a moving image shooting mode capable of shooting asmooth moving image and a moving image shooting mode in which rollingdistortion generally produced in a CMOS-type imaging element is notgenerated. Therefore, in the present embodiment, either a first movingimage/still image shooting mode capable of shooting a smooth movingimage or a second moving image/still image shooting mode capable ofshooting a moving image without rolling distortion is selected dependingon the shutter speed of the first moving image. Referring to theflowchart of FIG. 34, shooting operation in the dual image mode will bedescribed below.

First, in step S501, the system control CPU 178 as a control unit of theimaging device checks on a moving image/still image shooting mode set bya person who performs shooting. When checking that the shooting mode isthe dual image mode to shoot the first moving image and the secondmoving image at the same time, the system control CPU 178 proceeds tostep S502.

Then, in step S502, the system control CPU 178 checks on a set shootingperiod of the second moving image.

Then, in step S503, the system control CPU 178 checks on a shutter speed(still image shutter speed) of the first moving image set by the personwho performs shooting.

Then, in step S504, the system control CPU 178 determines whether theset shutter speed of the first moving image is faster than apredetermined value. When determining that the shutter speed of thefirst moving image is set to a shutter speed faster than a predeterminedshutter speed Tth to obtain an image having the stop motion effect on anobject moving fast (yes), the system control CPU 178 proceeds to stepS505. In step S505, the system control CPU 178 sets the movingimage/still image shooting mode to the second moving image/still imageshooting mode (undistorted moving image shooting mode) in which rollingdistortion is not generated, and proceeds to step S507.

On the other hand, when determining that the shutter speed of the firstmoving image is set to a shutter speed slower than the predeterminedshutter speed Tth (no), the system control CPU 178 proceeds to stepS506. In step S506, the system control CPU 178 sets the movingimage/still image shooting mode to the first moving image/still imageshooting mode (smooth moving image shooting mode) capable of shooting asmooth moving image, and proceeds to step S507.

When the moving image/still image shooting mode is set in step S505 orstep S506, the system control CPU 178 sets, in step S507, a controlmethod for the imaging element 184 according to the set movingimage/still image shooting mode. The control methods for the imagingelement 184 in the first moving image/still image shooting mode and thesecond moving image/still image shooting mode will be described later.

Then, in step S508, the system control CPU 178 checks on the state ofthe switch MV 155 as a button used to start and stop moving imageshooting through the switch input unit 179 to determine whether to startshooting. When the start of moving image shooting is not instructed atthe switch MV 155 (no), the system control CPU 178 returns to step S501to repeat the procedure from checking on the moving image/still imageshooting mode. On the other hand, when the start of moving imageshooting is instructed at the switch MV 155 (yes), the system controlCPU 178 proceeds to step S509.

In step S509, the system control CPU 178 controls the aperture 181 ofthe photographing optical system 152 through the aperture control unit182 based on AE information on images captured before then and the setshutter speed of the first moving image.

Then, in step S510, the system control CPU 178 drives the imagingelement 184 through the timing generation unit 189 to perform shooting.In the present embodiment, since the shooting mode is the dual imagemode to shoot the first moving image and the second moving image at thesame time, the shooting operation is performed by the switch MV 155 asthe button to start and stop moving image shooting. The shootingoperation is performed according to the control method for the imagingelement 184 set in step S507. The control method for the imaging element184 will be described later.

Then, in step S511, the system control CPU 178 checks on the state ofthe switch MV 155 as the button to start and stop moving image shootingthrough the switch input unit 179 to determine whether the shooting iscompleted. When the switch MV 155 is set in a shooting state (no), thesystem control CPU 178 returns to step S509 to continue shooting. On theother hand, when the switch MV 155 is set in a shooting stopped state(yes), the system control CPU 178 proceeds to step S512 to stopshooting.

FIG. 35 is a chart for describing the accumulation and readout timingsof the imaging element 184 in the imaging device of the presentembodiment when the first moving image and the second moving image areshot at the same time in the first moving image/still image shootingmode capable of shooting a smooth moving image. The term “accumulation”here means operation for transferring and accumulating charge generatedin the photodiode 600 to and in the signal holding units 607A, 607B. Theterm “readout” means operation for outputting signals based on thecharges, held in the signal holding units 607A, 607B, to the outside ofthe imaging element 184 via the floating diffusion region 608.

In FIG. 35, the abscissa is expressed in time to illustrate a verticalsynchronization signal 650, a horizontal synchronization signal 651, astill image accumulation period 661, a still image transfer period 662,a still image readout period 665, a moving image accumulation period663, a moving image transfer period 664, and a moving image readoutperiod 666. Here, the still image accumulation period 661 indicates anaccumulation period of signal charge for the first moving image into thephotodiode 600. The still image transfer period 662 indicates a periodof transferring the signal charge for the first moving image from thephotodiode 600 to the signal holding unit 607A. The still image readoutperiod 665 indicates a readout period of the first moving image. Themoving image accumulation period 663 indicates an accumulation period ofsignal charge for the second moving image into the photodiode 600. Themoving image transfer period 664 indicates a period of transferring thesignal charge for the second moving image from the photodiode 600 to thesignal holding unit 607B. The moving image readout period 666 indicatesa readout period of the second moving image.

In this driving example, the first moving image and the second movingimage are read out during each cycle of the vertical synchronizationsignal 650. Further, timings of 16 rows are illustrated in FIG. 35 fordescriptive purposes, but the actual imaging element 184 has thousandsof rows. In FIG. 35, the final row is the m-th row.

The first moving image is generated based on signal charge generatedduring one accumulation period (still image accumulation period 661)performed simultaneously in all rows during each cycle (time Tf) of thevertical synchronization signal 650. The second moving image isgenerated based on signal charge obtained by adding up signal chargesrespectively generated during accumulation periods (moving imageaccumulation periods 663) divided by the number of Np times (where Np isan integer of 2 or more (Np>1)). Np as the number of accumulationperiods of the second moving image performed during one shooting periodis, for example, 16 times, and these accumulation periods are performedat equal time intervals. The interval (time Tf) of the verticalsynchronization signal 650 is 1/60 second, which approximatelycorresponds to a period during which Np times of accumulation periods ofthe second moving image are performed in the first moving image/stillimage shooting mode. The accumulation of the first moving image isperformed during the readout of the second moving image (moving imagereadout period 666) in one shooting period.

This enables shooting of the first moving image and the second movingimage at the same time. An image having no blur can also be acquired asthe first moving image at a short accumulation time intended by theperson who performs shooting. Further, Np times of accumulation periodsperformed at equal time intervals virtually mean one long accumulationperiod from the start time of the first accumulation period to the endtime of the Np-th accumulation period. Therefore, a smooth image withless jerkiness can be acquired as the second moving image.

In FIG. 35, the accumulation period of the first moving image (stillimage accumulation period 661) is set to a time corresponding to ashutter speed T1 set by the person who performs shooting. In thisdriving example, the shutter speed T1 is set to 1/500 second. Theaccumulation period of the first moving image is set to be performedsimultaneously in all rows and be completed immediately before the startof readout of the first moving image in the first row (still imagereadout period 665). The end time of the accumulation period of thefirst moving image is a time after a lapse of time Ta from the verticalsynchronization signal 650. The time Ta is set to be half or less of theinterval Tf of the vertical synchronization signal 650. Since the endtime of the accumulation period of the first moving image (still imageaccumulation period 661) is the same in all rows, the start time of theaccumulation period of the first moving image with respect to thevertical synchronization signal 650 is set according to the shutterspeed T1 of the first moving image.

On the other hand, the accumulation period of the second moving image(moving image accumulation period 663) is performed plural times atequal time intervals during each cycle. In this driving example, thetime interval is set to complete the accumulation period divided into 16times immediately before the start of the readout of each row (movingimage readout period 666). The time interval of the accumulation periodof the second moving image may be set to be a multiple of an integer forthe interval Th of the horizontal synchronizing signal 651. Thus, theaccumulation timing of the second moving image in each row is the sameas that in the other rows. In FIG. 35, the time interval of theaccumulation period of the second moving image is illustrated to betwice the interval Th of the horizontal synchronization signal 651 fordescriptive purposes. When the number of rows of the imaging element 184is denoted by m, and the number of accumulations of the second movingimage during each cycle is denoted by Np, the time interval of theaccumulation period of the second moving image is generally set to avalue obtained by multiplying an integer not exceeding m/Np by theinterval Th of the horizontal synchronization signal 651.

Further, one accumulation time of the second moving image is set toT1/Np (= 1/8000 second). The start time of the accumulation period ofthe second moving image in each row is fixed with respect to thevertical synchronization signal 650. The end time of one accumulationperiod of the second moving image is set with respect to the verticalsynchronization signal 650 depending on the still image shutter speed T1set by the person who performs shooting.

In FIG. 35, since the accumulation time (T1) of the first moving imageis long, the number of accumulations Np of the second moving imageduring each cycle is 14 times. Therefore, the second moving imagegenerated during one shooting period is corrected using the first movingimage generated during the same shooting period.

Referring next to a timing chart of FIG. 36, an example of the controlmethod for the imaging element 184 during the shooting period startingat time t1 in FIG. 35 will be described. Time t1 at which a verticalsynchronization signal ϕV rises in FIG. 36 is the same as time t1 atwhich the vertical synchronization signal 650 rises in FIG. 35.

It is assumed here that the imaging element 184 has m rows of pixels inthe vertical direction. In FIG. 36, the timings of the first row and them-th row as the final row are illustrated among the m rows. In FIG. 36,a signal ϕV is the vertical synchronization signal, and a signal ϕH isthe horizontal synchronization signal.

First, at time t1, the vertical synchronization signal ϕV and thehorizontal synchronization signal ϕH supplied from the timing generationunit 189 are changed from the low level to the high level.

Then, at time t2 synchronized with the change of the verticalsynchronization signal ϕV to the high level, a reset pulse ϕRES(1) forthe first row supplied from the vertical scanning circuit 307 is changedfrom the high level to the low level. This causes the reset transistor604 of each pixel 303 in the first row to be turned off to release thereset state of the floating diffusion region 608. Simultaneously, aselect pulse ϕSEL(1) for the first row supplied from the verticalscanning circuit 307 is changed from the low level to the high level.This causes the select transistor 606 of each pixel 303 in the first rowto be turned on to enable the readout of an image signal from each pixel303 in the first row.

Then, at time t3, a transfer pulse ϕTX2B(1) for the first row suppliedfrom the vertical scanning circuit 307 is changed from the low level tothe high level. This causes the transfer transistor 602B of each pixel303 in the first row to be turned on to transfer, to the floatingdiffusion region 608, signal charge of the second moving imageaccumulated in the signal holding unit 607B during the previous shootingperiod (a shooting period completed at time t1). As a result, a signalcorresponding to a change in the potential of the floating diffusionregion 608 is read out into the signal output line 623 via the amplifiertransistor 605 and the select transistor 606. The signal read out intothe signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the secondmoving image of each pixel in the first row (corresponding to the movingimage readout period 666 in FIG. 35).

Then, at time t4, a transfer pulse ϕTX2B(1) for the first row andtransfer pulses ϕTX2A (ϕTX2A(1), ϕTX2A(m)) for all rows supplied fromthe vertical scanning circuit 307 are changed from the low level to thehigh level. This causes the transfer transistor 602B of each pixel 303in the first row and the transfer transistors 602A of the pixels 303 inall rows to be turned on. At this time, the reset pulses ϕRES (ϕRES(1),ϕRES(m)) in all rows are already changed to the high level, and hencethe reset transistors 604 are in the on-state. Thus, the floatingdiffusion regions 608 of the pixels 303 in all rows, the signal holdingunits 607A of the pixels 303 in all rows, and the signal holding unit607B of each pixel 303 in the first row are reset. At this time, theselect pulse ϕSEL(1) in the first row is also changed to the low level,and each pixel 303 in the first row is returned to an unselected state.

Then, at time t5, transfer pulses ϕTX3 (ϕTX3(1), ϕTX3(m)) for all rowssupplied from the vertical scanning circuit 307 are changed from thehigh level to the low level. This causes the transfer transistors 603 inall rows to be turned off to release the reset of the photodiodes 600 ofthe pixels 303 in all rows so as to start the accumulation of signalcharge of the second moving image in the photodiodes 600 of the pixels303 in all rows (corresponding to the moving image accumulation period663 in FIG. 35).

Here, a time interval Tb between time t1, at which the verticalsynchronization signal ϕV becomes the high level, and time t5, at whichthe accumulation of signal charge of the second moving image in thephotodiodes 600 of the pixels 303 in all rows is started, is fixed.

Note that the start of the accumulation period of the first row of thesecond moving image at time t5 in FIG. 36 represents the start of theaccumulation period of the second moving image in the shooting periodfrom time t1 in FIG. 35. Further, the start of the accumulation periodof the m-th row of the second moving image at time t5 represents thestart of the accumulation period of the second moving image in theshooting period before time t1 in FIG. 35.

Then, immediately before time t7, transfer pulses ϕTX1B (ϕTX1B(1),ϕTX1B(m)) for all rows supplied from the vertical scanning circuit 307are changed from the low level to the high level. This causes thetransfer transistors 601B of the pixels 303 in all rows to be turned onto transfer, to the signal holding units 607B, the signal chargesaccumulated in the photodiodes 600 of the pixels 303 in all rows(corresponding to the moving image transfer period 664 in FIG. 35).

Then, at time t7, the transfer pulses ϕTX1B (ϕTX1B(1), ϕTX1B(m)) for allrows are changed from the high level to the low level. This causes thetransfer transistors 601B of the pixels 303 in all rows to be turned offto complete the transfer of the signal charges accumulated in thephotodiodes 600 to the signal holding units 607B.

A period from time t5 to time t7 corresponds to the accumulation time(=T1/16) in each of the Np accumulation periods of the second movingimage.

Similarly, at time t7, the transfer pulses ϕTX3 (ϕTX3(1), ϕTX3(m)) forall rows are changed from the low level to the high level. This causesthe transfer transistors 603 of the pixels 303 in all rows to be turnedon to put the photodiodes 600 of the pixels 303 in all rows into thereset state.

The second accumulation period of the second moving image is started attime t8 after a lapse of the time twice the interval Th of thehorizontal synchronization signal φH from time t5 at which the firstaccumulation period of the second moving image in the shooting periodstarting at time t1 is started.

Since the operation of the second accumulation period of the secondmoving image starting at time t8 and ending at time t10 is the same asthe operation of the first accumulation period of the second movingimage starting at time t5 and ending at time t7 as mentioned above, thedescription thereof will be omitted.

Here, in the operation of the first and the second accumulation periodsof the second moving image, signal charges of the second moving imagegenerated during these two accumulation periods are added up and held inthe signal holding unit 607B.

Then, during a period from time t10 to time t11, the third to fifthaccumulation periods of the second moving image are performed in thesame manner as the period from time t5 to time t7 as mentioned above.

Then, the sixth accumulation period of the second moving image isstarted at time t11. Here, the start time t11 of the sixth accumulationperiod of the second moving image is set to the time after a lapse ofthe time T (=6×2×Th+Tb) from time t1 at which the verticalsynchronization signal ϕV becomes the high level. Here, Th denotes thetime interval of the horizontal synchronization signal ϕH, and Tbdenotes a time interval between time t1 at which the verticalsynchronization signal ϕV becomes the high level and time t5 at whichthe first accumulation period of the second moving image is started inthe photodiode 600.

Since the operation of the sixth accumulation period of the secondmoving image starting at time t11 and ending at time t13 is the same asthe operation of the first accumulation period of the second movingimage starting at time t5 and ending at time t7 as mentioned above, thedescription thereof will be omitted.

Then, the accumulation period of the first moving image as the firstimage is started at time t14. In this driving example, the number ofaccumulation periods of the first moving image in one shooting period isonce. The start time of the readout period of the first moving image(corresponding to the still image readout period 665 in FIG. 35) withrespect to the vertical synchronization signal ϕV is fixed. Therefore,the end time of the accumulation period of the first moving image withrespect to the vertical synchronization signal ϕV is fixed to a timeafter a lapse of time Ta from the start time, and the accumulationperiod of the first moving image is set to be completed at time t19.Here, a time interval from time t1 to time t19 corresponds to time Ta inFIG. 35. The start time of the accumulation period of the first movingimage is controlled based on the shutter speed T1 of the first movingimage set by the person who performs shooting.

At time t14 back by time T1 from time t19 as the end time of theaccumulation period of the first moving image, the transfer pulses ϕTX3(ϕTX3(1), ϕTX3(m)) for all rows are changed from the high level to thelow level. This causes the transfer transistors 603 of the pixels 303 inall rows to be turned off to release the reset of the photodiodes 600 ofthe pixels 303 in all rows. Then, the accumulation period of signalcharge of the first moving image in the photodiodes 600 of the pixels303 in all rows is started (corresponding to the still imageaccumulation period 661 in FIG. 35).

Further, during the accumulation period of signal charge of the firstmoving image, the readout period of the m-th row of the second movingimage in the previous shooting period that ends at time t1 is completed.

First, at time t15, the reset pulse ϕRES(m) for the m-th row suppliedfrom the vertical scanning circuit 307 is changed from the high level tothe low level. This causes the reset transistor 604 of each pixel 303 inthe m-th row to be turned off to release the reset state of the floatingdiffusion region 608. Simultaneously, a select pulse ϕSEL(m) for them-th row supplied from the vertical scanning circuit 307 is changed fromthe low level to the high level. This causes the select transistor 606of each pixel 303 in the m-th row to be turned on to enable the readoutof the image signal from each pixel 303 in the m-th row.

Then, at time t16, a transfer pulse ϕTX2B(m) for the m-th row is changedfrom the low level to the high level. This causes the transfertransistor 602B of each pixel 303 in the m-th row to be turned on totransfer, to the floating diffusion region 608, the signal charge of thesecond moving image accumulated in the signal holding unit 607B duringthe previous shooting period that ends at time t1. As a result, a signalcorresponding to a change in the potential of the floating diffusionregion 608 is read out into the signal output line 623 via the amplifiertransistor 605 and the select transistor 606. The signal read out intothe signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the secondmoving image of each pixel in the m-th row (corresponding to the movingimage readout period 666 in FIG. 35).

Thus, the readout of the second moving image in the previous shootingperiod that ends at time t1 is completed. Next, the readout of the firstmoving image in the shooting period that starts at time t1 is performed(corresponding to the still image readout period 665 in FIG. 35).

Then, at time t17, the transfer pulse ϕTX2B(m) for the m-th row ischanged from the low level to the high level. This causes the transfertransistor 602B of each pixel 303 in the m-th row to be turned on. Atthis time, the reset pulse ϕRES(m) in the m-th row is already changed tothe high level, and hence the reset transistor 604 is in the on-state.Thus, the floating diffusion region 608 of each pixel 303 in the m-throw, and the signal holding unit 607B of each pixel 303 in the m-th roware reset. At this time, the select pulse ϕSEL(m) in the m-th row isalso changed to the low level, and each pixel in the m-th row isreturned to the unselected state.

Then, at time t18, the reset pulse ϕRES(1) for the first row is changedfrom the high level to the low level. This causes the reset transistor604 of each pixel 303 in the first row to be turned off to release thereset of the floating diffusion region 608. Simultaneously, the selectpulse ϕSEL(1) for the first row is changed from the low level to thehigh level. This causes the select transistor 606 of each pixel 303 inthe first row to be turned on to enable the readout of an image signalfrom each pixel 303 in the first row.

Then, immediately before time t19, transfer pulses ϕTX1A (ϕTX1A(1),ϕTX1A(m)) for all rows supplied from the vertical scanning circuit 307are changed from the low level to the high level. This causes thetransfer transistors 601A of the pixels 303 in all rows to be turned onto transfer, to the signal holding units 607A, the signal chargesaccumulated in the photodiodes 600 of the pixels 303 in all rows(corresponding to the still image transfer period 662 in FIG. 35).

At time t19, the transfer pulses ϕTX1A (ϕTX1A(1), ϕTX1A(m)) for all rowsare changed from the high level to the low level. This causes thetransfer transistors 601A of the pixels 303 in all rows to be turned offto complete the transfer of the signal charges accumulated in thephotodiodes 600 of the pixels 303 in all rows to the signal holdingunits 607A.

A period from time t14 to time t19 corresponds to the accumulation time(T1) of the first moving image in the shooting period that starts attime t1. In this driving example, since the number of accumulationperiods of the first moving image in one shooting period is once, theaccumulation time of the first moving image is the same as the timecorresponding to the accumulation period.

Then, at time t20, the transfer pulse ϕTX2A(1) for the first row ischanged from the low level to the high level. This causes the transfertransistor 602A of each pixel 303 in the first row to be turned on totransfer, to the floating diffusion region 608, the signal chargeaccumulated in the signal holding unit 607A of each pixel 303 in thefirst row. As a result, a signal corresponding to a change in thepotential of the floating diffusion region 608 is read out into thesignal output line 623 via the amplifier transistor 605 and the selecttransistor 606 of each pixel 303 in the first row. The signal read outinto the signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the firstmoving image of each pixel in the first row (corresponding to the stillimage readout period 665 in FIG. 35).

Then, the seventh accumulation period of the second moving image isstarted at time t21. Here, the start time t21 of the seventhaccumulation period of the second moving image is set to a time after alapse of time T (=(7+2)×2×Th+Tb) from time t1 at which the verticalsynchronization signal ϕV becomes the high level. In this drivingexample, two accumulation periods of the second moving image overlap theaccumulation period of the first moving image (corresponding to thestill image accumulation period 661 in FIG. 35). Therefore, the starttime t21 of the seventh accumulation period of the second moving imageis equivalent to the start time of the ninth accumulation period of thesecond moving image in the shooting period that starts at time t1.

Since the operation of the seventh accumulation period of the secondmoving image starting at time t21 and ending at time t23 is the same asthe operation of the first accumulation period of the second movingimage starting at time t5 and ending at time t7 as mentioned above, thedescription thereof will be omitted.

Then, during a period from time t23 to time t24, the eighth tothirteenth accumulation periods of the second moving image are performedin the same manner as the period from time t5 to time t7 as mentionedabove.

Then, the final fourteenth accumulation period of the second movingimage in the shooting period that starts at time t1 is started at timet24. Here, the start time t24 of the fourteenth accumulation period ofthe second moving image is set to a time after a lapse of time T(=(14+2)×2×Th+Tb) from time t1 at which the vertical synchronizationsignal ϕV becomes the high level.

Since the operation of the fourteenth accumulation period of the secondmoving image starting at time t24 and ending at time t26 is the same asthe operation of the first accumulation period of the second movingimage starting at time t5 and ending at time t7 as mentioned above, thedescription thereof will be omitted. In the shooting mode, the period toperform the Np accumulation periods of the second moving image is aperiod from time t5 to time t26.

Then, at time t27, the reset pulse ϕRES(m) for the m-th row is changedfrom the high level to the low level. This causes the reset transistor604 of each pixel 303 in the m-th row to be turned off to release thereset state of the floating diffusion region 608. Simultaneously, theselect pulse ϕSEL(m) for the m-th row is changed from the low level tothe high level. This causes the select transistor 606 of each pixel 303in the m-th row to be turned on to enable the readout for the imagesignal from each pixel 303 in the m-th row.

Then, at time t28, the transfer pulse ϕTX2A(m) for the m-th row ischanged from the low level to the high level. This causes the transfertransistor 602A of each pixel 303 in the m-th row to be turned on totransfer, to the floating diffusion region 608, the signal charge of thefirst moving image accumulated in the signal holding unit 607A of eachpixel 303 in the m-th row. As a result, a signal corresponding to achange in the potential of the floating diffusion region 608 is read outinto the signal output line 623 via the amplifier transistor 605 and theselect transistor 606 of each pixel 303 in the m-th row. The signal readout into the signal output line 623 is supplied to an unillustratedreadout circuit, and output to the outside as an image signal of thefirst moving image of each pixel in the m-th row (corresponding to thestill image readout period 665 in FIG. 35).

Then, at time t29, the vertical synchronization signal ϕV supplied fromthe timing generation unit 189 is changed from the low level to the highlevel to start the next shooting period.

As described above, in the first moving image/still image shooting mode,the end time of the accumulation period of the first moving image isfixed with respect to the vertical synchronization signal, and the starttime of the accumulation periods of the second moving image performedplural times in one shooting period is fixed with respect to thevertical synchronization signal. This enables the readout of the firstmoving image and the second moving image within the same shootingperiod.

Therefore, when the shutter speed T1 of the first moving image is slowerthan the predetermined shutter speed Tth, the first moving image shortin accumulation time and having no blur, and the second moving imagelong in accumulation period and with less jerkiness can be shot in oneshooting period at the same time.

FIG. 37 is a chart for describing the accumulation and readout timingsof the imaging element 184 in the imaging device of the presentembodiment when the first moving image and the second moving image areshot at the same time in the second moving image/still image shootingmode capable of shooting a moving image without rolling distortion. Theterm “accumulation” here means operation for transferring andaccumulating charge generated in the photodiode 600 to and in the signalholding units 607A, 607B. The term “readout” means operation foroutputting signals based on the electric charges, held in the signalholding units 607A, 607B, to the outside of the imaging element 184 viathe floating diffusion region 608.

In FIG. 37, the abscissa is expressed in time to illustrate a verticalsynchronization signal 650, a horizontal synchronization signal 651, astill image accumulation period 661, a still image transfer period 662,a still image readout period 665, a moving image accumulation period663, a moving image transfer period 664, and a moving image readoutperiod 666. In this driving example, the first moving image and thesecond moving image are read out during each cycle of the verticalsynchronization signal 650. Further, timings of 16 rows are illustratedin FIG. 37 for descriptive purposes, but the actual imaging element 184has thousands of rows. In FIG. 37, the final row is the m-th row.

The first moving image is generated based on signal charge generatedduring one accumulation period (still image accumulation period 661)performed simultaneously in all rows during each cycle (time Tf) of thevertical synchronization signal 650. The second moving image isgenerated based on signal charge obtained by adding up signal chargesrespectively generated during accumulation periods (moving imageaccumulation periods 663) divided by the number of Np times (where Np isan integer of 2 or more (Np>1)). Np as the number of accumulationperiods of the second moving image performed during one shooting periodis, for example, eight times, and these accumulation periods areperformed in all rows at equal time intervals during the readout period(still image readout period 665) of the first moving image. The interval(time Tf) of the vertical synchronization signal 650 corresponds to theframe rate of the moving image, which is 1/60 second in this drivingexample. Further, the accumulation of the first moving image isperformed during the readout of the second moving image (moving imagereadout period 666) in one shooting period.

This enables shooting of the first moving image and the second movingimage at the same time. An image having no blur can also be acquired asthe first moving image at a short accumulation time intended by theperson who performs shooting. Further, Np times of accumulation periodsperformed at equal time intervals virtually mean one long accumulationperiod from the start time of the first accumulation period to the endtime of the Np-th accumulation period. Therefore, an image with lessjerkiness and without rolling distortion can be acquired as the secondmoving image.

In the previous shooting period that ends at time t51 in FIG. 37, theaccumulation period of the first moving image (still image accumulationperiod 661) is set to a time corresponding to a shutter speed T2 set bythe person who performs shooting. In this driving example, the shutterspeed T2 is set to 1/2000 second. The center time of the accumulationperiod of the first moving image is the same in all rows (a time after alapse of time Tc from the vertical synchronization signal 650), which isso set that the accumulation period will be completed before the readoutperiod of the first row of the first moving image (still image readoutperiod 665). Here, since the time Tc up to the center time of theaccumulation period of the first moving image is the same in all rows,the start time and end time of the accumulation period of the firstmoving image with respect to the vertical synchronization signal 650 areset depending on the shutter speed T2 of the first moving image. Thetime Tc up to the center time of the accumulation period of the firstmoving image is set to be the center of the readout period of the secondmoving image (moving image readout period 666), which is set to be about¼ of the time Tf corresponding to the interval of the verticalsynchronization signal 650.

On the other hand, the accumulation period of the second moving image(moving image accumulation period 663) is performed plural times atequal time intervals during the readout period of the first moving image(still image readout period 665). In this driving example, the timeinterval is set to complete the accumulation period divided into eighttimes immediately before the start of the readout period of the firstrow of the second moving image (moving image readout period 666). Thetime interval of the accumulation period of the second moving image isset to be a multiple of an integer for the interval Th of the horizontalsynchronization signal 651. Thus, the Np accumulation periods of allrows of the second moving image become the same. In FIG. 37, the timeinterval of the accumulation period of the second moving image isillustrated to be twice the interval Th of the horizontalsynchronization signal 651 for descriptive purposes. When the number ofrows of the imaging element 184 is denoted by m, and the number ofaccumulations of the second moving image during each cycle is denoted byNp, the time interval of the accumulation period of the second movingimage is generally set to a value obtained by multiplying an integer notexceeding m/Np by the interval Th of the horizontal synchronizing signal651.

Further, one accumulation time of the second moving image is set toT2/Np (= 1/16000 second). The start time of the accumulation period ofthe second moving image in all rows is fixed with respect to thevertical synchronization signal 650. The end time of one accumulationperiod of the second moving image is set with respect to the verticalsynchronization signal 650 depending on the shutter speed T2 of thefirst moving image set by the person who performs shooting.

It is also effective that the dead time of the second moving imagegenerated during the previous shooting period that ends at time t51 iscorrected using the first moving image generated in this shootingperiod.

Thus, the accumulation period of the second moving image is performed inall rows at the same timing during the readout period of a still image(still image readout period 665) so that a moving image without rollingdistortion can be acquired.

Referring next to a timing chart of FIG. 38, an example of the controlmethod for the imaging element 184 in a shooting period starting at timet51 in FIG. 37 will be described. Time t51 at which a verticalsynchronization signal ϕV rises in FIG. 38 is the same as time t51 atwhich the vertical synchronization signal ϕV 650 rises in FIG. 37.

It is assumed here that the imaging element 184 has m rows of pixels inthe vertical direction. In FIG. 38, the timings of the first row and them-th row as the final row are illustrated among the m rows. In FIG. 38,a signal ϕVV is the vertical synchronization signal, and a signal ϕVH isthe horizontal synchronization signal.

First, at time t51, the vertical synchronization signal ϕV and thehorizontal synchronization signal ϕH supplied from the timing generationunit 189 are changed from the low level to the high level.

Then, at time t52 synchronized with the change of the verticalsynchronization signal ϕV to the high level, a reset pulse ϕRES(1) forthe first row supplied from the vertical scanning circuit 307 is changedfrom the high level to the low level. This causes the reset transistor604 of each pixel 303 in the first row to be turned off to release thereset state of the floating diffusion region 608. Simultaneously, aselect pulse ϕSEL(1) for the first row supplied from the verticalscanning circuit 307 is changed from the low level to the high level.This causes the select transistor 606 of each pixel 303 in the first rowto be turned on to enable the readout of an image signal from each pixel303 in the first row.

Then, at time t53, a transfer pulse ϕTX2B(1) for the first row suppliedfrom the vertical scanning circuit 307 is changed from the low level tothe high level. This causes the transfer transistor 602B of each pixel303 in the first row to be turned on to transfer, to the floatingdiffusion region 608, signal charge of the second moving imageaccumulated in the signal holding unit 607B during the previous shootingperiod (a shooting period completed at time t51). As a result, a signalcorresponding to a change in the potential of the floating diffusionregion 608 is read out into the signal output line 623 via the amplifiertransistor 605 and the select transistor 606. The signal read out intothe signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the secondmoving image of each pixel in the first row (corresponding to the movingimage readout period 666 in FIG. 37).

Then, at time t54, a transfer pulse ϕTX2B(1) for the first row andtransfer pulses ϕTX2A (ϕTX2A(1), ϕTX2A(m)) for all rows supplied fromthe vertical scanning circuit 307 are changed from the low level to thehigh level. This causes the transfer transistor 602B of each pixel 303in the first row and the transfer transistors 602A of the pixels 303 inall rows to be turned on. At this time, the reset pulses ϕRES (ϕRES(1),ϕRES(m)) in all rows are already changed to the high level, and hencethe reset transistors 604 are in the on-state. Thus, the floatingdiffusion regions 608 of the pixels 303 in all rows, the signal holdingunits 607A of the pixels 303 in all rows, and the signal holding unit607B of each pixel 303 in the first row are reset. At this time, theselect pulse ϕSEL(1) in the first row is also changed to the low level,and each pixel 303 in the first row is returned to the unselected state.

Then, the accumulation period of the first moving image is performedfrom time t55. In this driving example, the number of accumulationperiods of the first moving image in one shooting period is once. Thecenter time of the accumulation period of the first moving image is thesame in all rows (a time after a lapse of time Tc from the verticalsynchronization signal 650), which is so set that the accumulationperiod will be completed before the readout period of the first row ofthe first moving image (still image readout period 665). Here, since thetime Tc up to the center time of the accumulation period of the firstmoving image is the same in all rows, the start time and end time of theaccumulation period of the first moving image with respect to thevertical synchronization signal 650 are set depending on a shutter speedT2 of the first moving image set by the person who performs shooting.

At time t55 back by time T2/2 from time t56 as the center time of theaccumulation period of the first moving image, transfer pulses ϕTX3(ϕTX3(1), ϕTX3(m)) for all rows are changed from the high level to thelow level. This causes the transfer transistors 603 of the pixels 303 inall rows to be turned off to release the reset of the photodiodes 600 ofthe pixels 303 in all rows. Then, in the photodiodes 600 of the pixels303 in all rows, the accumulation period of signal charge of the firstmoving image is started (corresponding to the still image accumulationperiod 661 in FIG. 37). Here, a period from time t51 to time t56corresponds to time Tc in FIG. 37. Further, the accumulation of thesignal charge of the first moving image is completed before the end ofthe readout period of the second moving image in the m-th row during ashooting period until time t51 (corresponding to the moving imagereadout period 666 in FIG. 37).

Then, immediately before time t57, transfer pulses ϕTX1A (ϕTX1A(1),ϕTX1A(m)) for all rows supplied from the vertical scanning circuit 307are changed from the low level to the high level. This causes thetransfer transistors 601A of the pixels 303 in all rows to be turned onto transfer, to the signal holding units 607A, signal chargesaccumulated in the photodiodes 600 of the pixels 303 in all rows(corresponding to the still image transfer period 662 in FIG. 37).

Then, at time t57, the transfer pulses ϕTX1A (ϕTX1A(1), ϕTX1A(m)) forall rows are changed from the high level to the low level. This causesthe transfer transistors 601A of the pixels 303 in all rows to be turnedoff to complete the transfer of the signal charges accumulated in thephotodiodes 600 to the signal holding units 607A.

A period from time t55 to time t57 corresponds to the accumulation time(shutter speed T2) of the first moving image in the shooting period thatstarts at time t51 in FIG. 37. In this driving example, since the numberof accumulation periods of the first moving image in one shooting periodis once, the accumulation time of the first moving image in one shootingperiod is the same as the accumulation period.

Then, at time t58, a reset pulse ϕRES(m) for the m-th row is changedfrom the high level to the low level. This causes the reset transistor604 of each pixel 303 in the m-th row to be turned off to release thereset state of the floating diffusion region 608. Simultaneously, aselect pulse ϕSEL(m) for the m-th row is changed from the low level tothe high level. This causes the select transistor 606 of each pixel 303in the m-th row to be turned on to enable the readout of an image signalfrom each pixel 303 in the m-th row.

Then, at time t59, a transfer pulse ϕTX2B(m) for the m-th row suppliedfrom the vertical scanning circuit 307 is changed from the low level tothe high level. This causes the transfer transistor 602B of each pixel303 in the m-th row to be turned on to transfer, to the floatingdiffusion region 608, signal charge of the second moving imageaccumulated in the signal holding unit 607B during the previous shootingperiod (a shooting period until time t51 in FIG. 37). As a result, asignal corresponding to a change in the potential of the floatingdiffusion region 608 is read out into the signal output line 623 via theamplifier transistor 605 and the select transistor 606. The signal readout into the signal output line 623 is supplied to an unillustratedreadout circuit, and output to the outside as an image signal of thesecond moving image of each pixel in the m-th row (corresponding to themoving image readout period 666 in FIG. 37).

Then, at time t60, the transfer pulse ϕTX2B(m) for the m-th row ischanged from the low level to the high level. This causes the transfertransistor 602B of each pixel 303 in the m-th row to be turned on. Atthis time, the reset pulse ϕRES(m) for the m-th row is already changedto the high level, and hence the reset transistor 604 is in theon-state. Thus, the floating diffusion regions 608 of each pixel 303 inthe m-th row and the signal holding unit 607B of each pixel 303 in them-th row are reset. At this time, the select pulse ϕSEL(m) in the m-throw is also changed to the low level, and each pixel 303 in the m-th rowis returned to the unselected state.

When the readout of a moving image as the second image in the previousshooting period that ends at time t51 is completed, the readout of thefirst moving image in the shooting period that start at time t51(corresponding to the still image readout period 665 in FIG. 37) isstarted. Further, the accumulation of the second moving image(corresponding to the moving image accumulation period 663 in FIG. 37)is started.

At time t61, the reset pulse ϕRES(1) for the first row is changed fromthe high level to the low level. This causes the reset transistor 604 ofeach pixel 303 in the first row to be turned off to release the resetstate of the floating diffusion region 608. Simultaneously, the selectpulse ϕSEL(1) for the first row is changed from the low level to thehigh level. This causes the select transistor 606 of each pixel 303 inthe first row to be turned on to enable the readout of an image signalfrom each pixel 303 in the first row.

Then, at time t62, the transfer pulse ϕTX2A(1) for the first row ischanged from the low level to the high level. This causes the transfertransistor 602A of each pixel 303 in the first row to be turned on totransfer, to the floating diffusion region 608, signal chargeaccumulated in the signal holding unit 607A of each pixel 303 in thefirst row. As a result, a signal corresponding to a change in thepotential of the floating diffusion region 608 is read out into thesignal output line 623 via the amplifier transistor 605 and the selecttransistor 606 of each pixel 303 in the first row. The signal read outinto the signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the firstmoving image in the first row (corresponding to the still image readoutperiod 665 in FIG. 37).

Then, at time t63, the transfer pulses ϕTX3 (ϕTX3(1), ϕTX3(m)) for allrows are changed from the high level to the low level. This causes thetransfer transistors 603 of the pixels 303 in all rows to be turned offto release the reset of the photodiodes 600 of the pixels 303 in allrows so as to start the accumulation of signal charge in the photodiodes600 (corresponding to the moving image accumulation period 663 in FIG.37).

Here, a time interval Tb between time t51, at which the verticalsynchronization signal ϕV becomes the high level, and time t63, at whichthe accumulation of signal charge in the photodiodes 600 of the pixels303 in all rows is started, is fixed.

Then, immediately before time t64, transfer pulses ϕTX1B (ϕTX1B(1),ϕTX1B(m)) for all rows supplied from the vertical scanning circuit 307are changed from the low level to the high level. This causes thetransfer transistors 601B of the pixels 303 in all rows to be turned onto transfer, to the signal holding units 607B, the signal chargesaccumulated in the photodiodes 600 of the pixels 303 in all rowscorresponding to the moving image transfer period 664 in FIG. 37).

Then, at time t64, the transfer pulses ϕTX1B (ϕTX1B(1), ϕTX1B(m)) forall rows are changed from the high level to the low level. This causesthe transfer transistors 601B of the pixels 303 in all rows to be turnedoff to complete the transfer of the signal charges accumulated in thephotodiodes 600 to the signal holding units 607B.

A period from time t63 to time t64 corresponds to the accumulation time(=T2/8) in one accumulation period for the second moving image.

Similarly, at time t64, the transfer pulses ϕTX3 (ϕTX3(1), ϕTX3(m)) forall rows are changed from the low level to the high level. This causesthe transfer transistors 603 of the pixels 303 in all rows to be turnedon to put the photodiodes 600 of the pixels 303 in all rows into thereset state.

The second accumulation period of the second moving image is started attime t65 after a lapse of the time twice the interval Th of thehorizontal synchronization signal ϕH from the start time t63 of thefirst accumulation period of the second moving image in the shootingperiod that starts at time t51.

Since the operation of the second accumulation period of the secondmoving image starting at time t65 and ending at time t66 is the same asthe operation of the first accumulation period of the second movingimage starting at time t63 and ending at time t64 as mentioned above,the description thereof will be omitted.

Here, in the operation of the first and the second accumulation periodsof the second moving image, signal charges of the second moving imagegenerated during these two accumulation periods are added up and held inthe signal holding unit 607B.

Then, during a period from time t66 to time t67, the third to seventhaccumulation periods of the second moving image are performed in thesame manner as the period from time t63 to time t64 as mentioned above.

Then, the eighth accumulation period of the second moving image as thefinal period in one shooting period is started at time t67. Here, thestart time t67 of the eighth accumulation period of the second movingimage is set to the time after a lapse of the time T (=8×2×Th+Tb) fromtime t51 at which the vertical synchronization signal ϕV becomes thehigh level. Here, Th denotes the time interval of the horizontalsynchronization signal ϕH, and Tb denotes a time interval between timet51 at which the vertical synchronization signal ϕV becomes the highlevel and time t63 at which the first accumulation period of the secondmoving image is started in the photodiode 600.

Since the operation of the eighth accumulation period of the secondmoving image starting at time t67 and ending at time t68 is the same asthe operation of the first accumulation period of the second movingimage starting at time t63 and ending at time t64 as mentioned above,the description thereof will be omitted.

The period from time t63 to time t68 is the period of accumulatingsignal charge for the second moving image in the shooting mode, which isperformed during the readout period of the first moving image (a periodfrom time t62 to time t70).

At time t69, at which the accumulation period of the second moving imageis completed, the reset pulse ϕRES(m) for the m-th row is changed fromthe high level to the low level. This causes the reset transistor 604 ofeach pixel 303 in the m-th row to be turned off to release the resetstate of the floating diffusion region 608. Simultaneously, the selectpulse ϕSEL(m) for the m-th row is changed from the low level to the highlevel. This causes the select transistor 606 of each pixel 303 in them-th row to be turned on to enable the readout of an image signal fromeach pixel 303 in the m-th row.

Then, at time t70, the transfer pulse ϕTX2A(m) for the m-th row ischanged from the low level to the high level. This causes the transfertransistor 602A of each pixel 303 in the m-th row to be turned on totransfer, to the floating diffusion region 608, the signal chargeaccumulated in the signal holding unit 607A of each pixel 303 in them-th row. As a result, a signal corresponding to a change in thepotential of the floating diffusion region 608 is read out into thesignal output line 623 via the amplifier transistor 605 and the selecttransistor 606 of each pixel 303 in the m-th row. The signal read outinto the signal output line 623 is supplied to an unillustrated readoutcircuit, and output to the outside as an image signal of the firstmoving image of each pixel in the m-th row (corresponding to the stillimage readout period 665 in FIG. 37).

Then, at time t71, the vertical synchronization signal ϕV supplied fromthe timing generation unit 189 is changed from the low level to the highlevel to start the next shooting period.

As described above, in the second moving image/still image shootingmode, the accumulation period of the second moving image is performed inall rows at the same timing during readout period of the first movingimage (still image readout period 665). Thus, a moving image withoutrolling distortion can be acquired. Further, since the accumulationperiod of the second moving image is set longer than the accumulationperiod of the first moving image, an image with less jerkiness can beacquired.

As described above, according to the present embodiment, “picture A”having the stop motion effect and “picture B” with less jerkiness can beacquired at the same time. The image presentation method as illustratedin the first embodiment can be used for two moving images difference incharacteristics to provide an image suitable for viewing of both ofmoving image/still image when two or more images are shot at the sametime and viewed using the single imaging element 184.

Alternative Embodiments

The present invention is not limited to the aforementioned exemplaryembodiments, and various modifications can be made.

For example, the configuration of the imaging device described in theaforementioned embodiments is just an example, and an imaging device towhich the present invention can be applied is not limited to theconfiguration illustrated in FIG. 1A to FIG. 2. Further, the circuitconfiguration of each unit of the imaging element is not limited to theconfiguration illustrated in FIG. 3, FIG. 8, FIG. 11, FIG. 32, or thelike.

Further, in the above first embodiment, the example of performingcrosstalk correction on “picture A” and “picture B” is illustrated asthe preferred mode, but the crosstalk correction is not necessarilyrequired.

Further, in the above first embodiment, the example of shooting “pictureA” and “picture B” at the same frame rate is illustrated, “picture A”and “picture B” are not necessarily required to be at the same framerate. In this case, for example, at least one of plural frames of“picture A” shot within one frame period of “picture B” can beassociated with the frame of “picture B.”

Further, in the above third embodiment, the accumulation period of thefirst moving image is performed once, and the accumulation period of thesecond moving image is performed sixteen times or eight times. However,the number of accumulation periods is selected appropriately accordingto the shooting conditions and the like, and not limited thereto. Forexample, the number of accumulations of the first moving image may be atleast once, or may be twice or more. Further, the number ofaccumulations of the second moving image may be at least twice or more.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blue-ray Disc(BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1-18. (canceled)
 19. A playback device comprising: a playback unit thatplaybacks first image files which are acquired based on a first signalcharge generated during a first accumulation time and second image fileswhich are acquired based on a second signal charge generated during asecond accumulation time and associated with the first image files;wherein the playback unit playbacks the first image files at a start ofplaybacking and playbacks the second image files when a pauseinstruction is received, and wherein the second accumulation timeoverlaps with at least part of the first accumulation time.
 20. Theplayback device according to claim 19, wherein the first image files arestill image data, and the second image files are moving image data. 21.The playback device according to claim 19, wherein the secondaccumulation time is relatively longer than the first accumulation time,and the second image files are recorded in synch with the first imagefiles during a synchronization period including the first accumulationtime.
 22. The playback device according to claim 19, wherein an exposurecondition for acquiring the first image files and an exposure conditionfor acquiring the second image files are different from each other. 23.The playback device according to claim 19, wherein the secondaccumulation time is relatively longer than the first accumulation time.24. The playback device according to claim 19, wherein the playback unitplaybacks the first image files and the second image files acquired byan imaging device including a plurality of pixels, each of the pluralityof pixels including a first photoelectric conversion unit and a secondphotoelectric conversion unit, and wherein the playback unit acquiresthe first image files based on the first signal charge generated by thefirst photoelectric conversion unit and the second image files based onthe second signal charge generated by the second photoelectricconversion unit.
 25. The playback device according to claim 19, whereinthe playback unit playbacks the first image files and the second imagefiles acquired by an imaging device including a plurality of pixels,each of the plurality of pixels including one photoelectric conversionunit, a first signal holding unit and a second signal holding unit, andwherein the playback unit acquires the first image files based onsignals generated by transferring a signal charge, generated by thephotoelectric conversion unit during one shooting period, to the firstsignal holding unit at least once, and the second image files based onsignals generated by transferring a signal charge, generated by thephotoelectric conversion unit during the one shooting period, to thesecond signal holding unit at least twice or more and adding up thesignal charges.
 26. The playback device according to claim 19, whereinthe first image files and the second image files are stored in storageon a network.
 27. The playback device according to claim 19, wherein thefirst image files and the second image files are associated to eachother with time codes therein.
 28. The playback device according toclaim 19, wherein the playback unit switches between playbacking of thefirst image files and playbacking of the second image files inaccordance with a frame rate.
 29. The playback device according to claim19, wherein the first image files and the second image files are storedin one file including synchronization information for synchronizing thefirst image files and the second image files frame by frame.